Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251026 02:56:16.860077 1624 test_util.cc:276] Using random seed: -315608212
2025-10-26T02:56:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T02:56:16Z Disabled control of system clock
I20251026 02:56:16.893858 1624 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
/tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.150.62:44525
--webserver_interface=127.1.150.62
--webserver_port=0
--builtin_ntp_servers=127.1.150.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251026 02:56:17.205642 1632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 02:56:17.205965 1632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 02:56:17.206035 1632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 02:56:17.214073 1632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 02:56:17.214202 1632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 02:56:17.214352 1632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 02:56:17.214435 1632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 02:56:17.225247 1632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.150.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.150.62:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.150.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 26 Oct 2025 02:43:21 UTC on 24a791456cd2
build id 8720
ASAN enabled
I20251026 02:56:17.226856 1632 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 02:56:17.228786 1632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 02:56:17.236898 1637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:17.236933 1638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:17.237201 1640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 02:56:17.237872 1632 server_base.cc:1047] running on GCE node
I20251026 02:56:17.238577 1632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 02:56:17.239956 1632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 02:56:17.241155 1632 hybrid_clock.cc:648] HybridClock initialized: now 1761447377241062 us; error 89 us; skew 500 ppm
I20251026 02:56:17.241465 1632 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 02:56:17.243501 1632 webserver.cc:492] Webserver started at http://127.1.150.62:37051/ using document root <none> and password file <none>
I20251026 02:56:17.244272 1632 fs_manager.cc:362] Metadata directory not provided
I20251026 02:56:17.244380 1632 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 02:56:17.244714 1632 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 02:56:17.246997 1632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/data/instance:
uuid: "209968db4dee47c49497a66c3ee750b7"
format_stamp: "Formatted at 2025-10-26 02:56:17 on dist-test-slave-h3w0"
I20251026 02:56:17.247695 1632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/wal/instance:
uuid: "209968db4dee47c49497a66c3ee750b7"
format_stamp: "Formatted at 2025-10-26 02:56:17 on dist-test-slave-h3w0"
I20251026 02:56:17.253330 1632 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20251026 02:56:17.256568 1646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:17.257848 1632 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20251026 02:56:17.258006 1632 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/data,/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/wal
uuid: "209968db4dee47c49497a66c3ee750b7"
format_stamp: "Formatted at 2025-10-26 02:56:17 on dist-test-slave-h3w0"
I20251026 02:56:17.258211 1632 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 02:56:17.277383 1632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 02:56:17.278313 1632 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 02:56:17.278577 1632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 02:56:17.297439 1632 rpc_server.cc:307] RPC server started. Bound to: 127.1.150.62:44525
I20251026 02:56:17.297446 1697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.150.62:44525 every 8 connection(s)
I20251026 02:56:17.298929 1632 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/data/info.pb
I20251026 02:56:17.300057 1624 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu as pid 1632
I20251026 02:56:17.300400 1624 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-0/wal/instance
I20251026 02:56:17.302913 1624 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
/tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.150.61:37655
--webserver_interface=127.1.150.61
--webserver_port=0
--builtin_ntp_servers=127.1.150.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251026 02:56:17.303383 1698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:17.309326 1698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } has no permanent_uuid. Determining permanent_uuid...
I20251026 02:56:17.326921 1698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } has no permanent_uuid. Determining permanent_uuid...
W20251026 02:56:17.327966 1650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.150.61:37655: connect: Connection refused (error 111)
W20251026 02:56:17.329594 1698 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.61:37655: Network error: Client connection negotiation failed: client connection to 127.1.150.61:37655: connect: Connection refused (error 111)
I20251026 02:56:17.378937 1698 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } attempt: 1
W20251026 02:56:17.380940 1698 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.61:37655: Network error: Client connection negotiation failed: client connection to 127.1.150.61:37655: connect: Connection refused (error 111)
I20251026 02:56:17.449193 1698 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } attempt: 2
W20251026 02:56:17.451032 1698 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.61:37655: Network error: Client connection negotiation failed: client connection to 127.1.150.61:37655: connect: Connection refused (error 111)
I20251026 02:56:17.542367 1698 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } attempt: 3
W20251026 02:56:17.544150 1698 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.61:37655: Network error: Client connection negotiation failed: client connection to 127.1.150.61:37655: connect: Connection refused (error 111)
W20251026 02:56:17.626279 1700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 02:56:17.626610 1700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 02:56:17.626729 1700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 02:56:17.634660 1700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 02:56:17.634760 1700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 02:56:17.634814 1700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 02:56:17.634878 1700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 02:56:17.644747 1700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.150.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.150.61:37655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.150.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 26 Oct 2025 02:43:21 UTC on 24a791456cd2
build id 8720
ASAN enabled
I20251026 02:56:17.646413 1700 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 02:56:17.648211 1700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 02:56:17.656335 1708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:17.656654 1709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:17.657064 1711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 02:56:17.657826 1700 server_base.cc:1047] running on GCE node
I20251026 02:56:17.658581 1700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 02:56:17.659950 1700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 02:56:17.661144 1700 hybrid_clock.cc:648] HybridClock initialized: now 1761447377661082 us; error 70 us; skew 500 ppm
I20251026 02:56:17.661481 1700 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 02:56:17.663676 1700 webserver.cc:492] Webserver started at http://127.1.150.61:34385/ using document root <none> and password file <none>
I20251026 02:56:17.664455 1700 fs_manager.cc:362] Metadata directory not provided
I20251026 02:56:17.664561 1700 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 02:56:17.664927 1700 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 02:56:17.667131 1700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/data/instance:
uuid: "647c0de4ce394caca75179fe11658fc8"
format_stamp: "Formatted at 2025-10-26 02:56:17 on dist-test-slave-h3w0"
I20251026 02:56:17.667869 1700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/wal/instance:
uuid: "647c0de4ce394caca75179fe11658fc8"
format_stamp: "Formatted at 2025-10-26 02:56:17 on dist-test-slave-h3w0"
I20251026 02:56:17.673238 1700 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251026 02:56:17.676519 1717 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:17.677863 1700 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251026 02:56:17.678031 1700 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/data,/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/wal
uuid: "647c0de4ce394caca75179fe11658fc8"
format_stamp: "Formatted at 2025-10-26 02:56:17 on dist-test-slave-h3w0"
I20251026 02:56:17.678239 1700 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 02:56:17.687500 1698 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } attempt: 4
W20251026 02:56:17.689301 1698 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.61:37655: Network error: Client connection negotiation failed: client connection to 127.1.150.61:37655: connect: Connection refused (error 111)
I20251026 02:56:17.689846 1700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 02:56:17.690608 1700 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 02:56:17.690816 1700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 02:56:17.710490 1700 rpc_server.cc:307] RPC server started. Bound to: 127.1.150.61:37655
I20251026 02:56:17.710525 1768 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.150.61:37655 every 8 connection(s)
I20251026 02:56:17.711936 1700 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/data/info.pb
I20251026 02:56:17.716163 1769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:17.719961 1624 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu as pid 1700
I20251026 02:56:17.720130 1624 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/wal/instance
I20251026 02:56:17.721168 1624 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
/tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.150.60:33729
--webserver_interface=127.1.150.60
--webserver_port=0
--builtin_ntp_servers=127.1.150.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251026 02:56:17.722288 1769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } has no permanent_uuid. Determining permanent_uuid...
I20251026 02:56:17.735230 1769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } has no permanent_uuid. Determining permanent_uuid...
I20251026 02:56:17.744729 1769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } has no permanent_uuid. Determining permanent_uuid...
W20251026 02:56:17.745870 1718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111)
W20251026 02:56:17.747336 1769 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.60:33729: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111)
I20251026 02:56:17.796645 1769 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } attempt: 1
W20251026 02:56:17.798571 1769 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.60:33729: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111)
I20251026 02:56:17.866825 1769 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } attempt: 2
W20251026 02:56:17.869077 1769 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.60:33729: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111)
I20251026 02:56:17.960348 1769 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } attempt: 3
W20251026 02:56:17.962327 1769 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.60:33729: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111)
I20251026 02:56:17.988540 1698 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } attempt: 5
I20251026 02:56:17.994405 1698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } has no permanent_uuid. Determining permanent_uuid...
W20251026 02:56:17.995910 1698 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.60:33729: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111)
I20251026 02:56:18.047137 1698 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } attempt: 1
W20251026 02:56:18.048990 1698 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.60:33729: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111)
W20251026 02:56:18.048943 1771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 02:56:18.049289 1771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 02:56:18.049402 1771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 02:56:18.057531 1771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 02:56:18.057649 1771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 02:56:18.057734 1771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 02:56:18.057816 1771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 02:56:18.067579 1771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.150.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.150.60:33729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.150.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 26 Oct 2025 02:43:21 UTC on 24a791456cd2
build id 8720
ASAN enabled
I20251026 02:56:18.069300 1771 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 02:56:18.071208 1771 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 02:56:18.079329 1779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:18.079331 1780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:18.079967 1782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 02:56:18.080611 1771 server_base.cc:1047] running on GCE node
I20251026 02:56:18.081388 1771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 02:56:18.082741 1771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 02:56:18.083938 1771 hybrid_clock.cc:648] HybridClock initialized: now 1761447378083893 us; error 43 us; skew 500 ppm
I20251026 02:56:18.084331 1771 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 02:56:18.086499 1771 webserver.cc:492] Webserver started at http://127.1.150.60:39671/ using document root <none> and password file <none>
I20251026 02:56:18.087250 1771 fs_manager.cc:362] Metadata directory not provided
I20251026 02:56:18.087354 1771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 02:56:18.087704 1771 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 02:56:18.090113 1771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/instance:
uuid: "751b719b75ce4bc2a88a19091230ecb7"
format_stamp: "Formatted at 2025-10-26 02:56:18 on dist-test-slave-h3w0"
I20251026 02:56:18.090924 1771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal/instance:
uuid: "751b719b75ce4bc2a88a19091230ecb7"
format_stamp: "Formatted at 2025-10-26 02:56:18 on dist-test-slave-h3w0"
I20251026 02:56:18.096841 1771 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20251026 02:56:18.100611 1788 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:18.102051 1771 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251026 02:56:18.102236 1771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data,/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
uuid: "751b719b75ce4bc2a88a19091230ecb7"
format_stamp: "Formatted at 2025-10-26 02:56:18 on dist-test-slave-h3w0"
I20251026 02:56:18.102473 1771 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 02:56:18.105563 1769 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } attempt: 4
W20251026 02:56:18.107339 1769 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.60:33729: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111)
I20251026 02:56:18.117221 1698 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } attempt: 2
W20251026 02:56:18.119004 1698 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.150.60:33729: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111)
I20251026 02:56:18.139626 1771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 02:56:18.140635 1771 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 02:56:18.140898 1771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 02:56:18.161708 1771 rpc_server.cc:307] RPC server started. Bound to: 127.1.150.60:33729
I20251026 02:56:18.161726 1839 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.150.60:33729 every 8 connection(s)
I20251026 02:56:18.163316 1771 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/info.pb
I20251026 02:56:18.167495 1840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:18.169516 1624 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu as pid 1771
I20251026 02:56:18.169716 1624 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal/instance
I20251026 02:56:18.173487 1840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } has no permanent_uuid. Determining permanent_uuid...
I20251026 02:56:18.197975 1840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } has no permanent_uuid. Determining permanent_uuid...
I20251026 02:56:18.209256 1840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } has no permanent_uuid. Determining permanent_uuid...
I20251026 02:56:18.219103 1840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Bootstrap starting.
I20251026 02:56:18.222942 1840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:18.223934 1840 log.cc:826] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Log is configured to *not* fsync() on all Append() calls
I20251026 02:56:18.225204 1698 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } attempt: 3
I20251026 02:56:18.226686 1840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: No bootstrap required, opened a new log
I20251026 02:56:18.234459 1840 raft_consensus.cc:359] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:18.234899 1840 raft_consensus.cc:385] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:18.235036 1840 raft_consensus.cc:740] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751b719b75ce4bc2a88a19091230ecb7, State: Initialized, Role: FOLLOWER
I20251026 02:56:18.235773 1840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:18.237646 1848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:18.238075 1848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:18.238689 1840 sys_catalog.cc:565] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 02:56:18.239114 1698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Bootstrap starting.
I20251026 02:56:18.243350 1698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:18.244449 1698 log.cc:826] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Log is configured to *not* fsync() on all Append() calls
I20251026 02:56:18.247238 1698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: No bootstrap required, opened a new log
W20251026 02:56:18.252125 1860 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 02:56:18.253005 1860 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 02:56:18.255067 1698 raft_consensus.cc:359] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:18.255426 1698 raft_consensus.cc:385] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:18.255573 1698 raft_consensus.cc:740] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209968db4dee47c49497a66c3ee750b7, State: Initialized, Role: FOLLOWER
I20251026 02:56:18.256203 1698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:18.250030 1771 master_runner.cc:186] Error getting master registration for 127.1.150.62:44525: OK, instance_id { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" instance_seqno: 1761447377292485 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251026 02:56:18.256546 1771 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 02:56:18.257161 1862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:18.257452 1862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:18.258359 1698 sys_catalog.cc:565] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: configured and running, proceeding with master startup.
W20251026 02:56:18.269866 1873 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 02:56:18.270097 1873 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 02:56:18.267390 1632 master_runner.cc:186] Error getting master registration for 127.1.150.61:37655: OK, instance_id { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" instance_seqno: 1761447377705108 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251026 02:56:18.274154 1632 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251026 02:56:18.281301 1652 tablet.cc:2378] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 02:56:18.393199 1794 tablet.cc:2378] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 02:56:18.406570 1769 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } attempt: 5
I20251026 02:56:18.419466 1769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8: Bootstrap starting.
I20251026 02:56:18.423802 1769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:18.425273 1769 log.cc:826] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8: Log is configured to *not* fsync() on all Append() calls
I20251026 02:56:18.428407 1769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8: No bootstrap required, opened a new log
I20251026 02:56:18.435978 1769 raft_consensus.cc:359] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:18.436379 1769 raft_consensus.cc:385] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:18.436508 1769 raft_consensus.cc:740] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 647c0de4ce394caca75179fe11658fc8, State: Initialized, Role: FOLLOWER
I20251026 02:56:18.437395 1769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:18.438544 1876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:18.438967 1876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:18.439873 1769 sys_catalog.cc:565] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [sys.catalog]: configured and running, proceeding with master startup.
W20251026 02:56:18.443076 1723 tablet.cc:2378] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 02:56:18.455647 1887 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251026 02:56:18.466121 1700 master_runner.cc:263] No leader master found from master 647c0de4ce394caca75179fe11658fc8
I20251026 02:56:18.466264 1700 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 02:56:18.504383 1862 raft_consensus.cc:493] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:18.504680 1862 raft_consensus.cc:515] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:18.506417 1862 leader_election.cc:290] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655), 751b719b75ce4bc2a88a19091230ecb7 (127.1.150.60:33729)
I20251026 02:56:18.507450 1815 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "209968db4dee47c49497a66c3ee750b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751b719b75ce4bc2a88a19091230ecb7" is_pre_election: true
I20251026 02:56:18.507404 1744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "209968db4dee47c49497a66c3ee750b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "647c0de4ce394caca75179fe11658fc8" is_pre_election: true
I20251026 02:56:18.507949 1815 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209968db4dee47c49497a66c3ee750b7 in term 0.
I20251026 02:56:18.507927 1744 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209968db4dee47c49497a66c3ee750b7 in term 0.
I20251026 02:56:18.508692 1650 leader_election.cc:304] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 209968db4dee47c49497a66c3ee750b7, 647c0de4ce394caca75179fe11658fc8, 751b719b75ce4bc2a88a19091230ecb7; no voters:
I20251026 02:56:18.509172 1862 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:18.509332 1862 raft_consensus.cc:493] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:18.509421 1862 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:18.512573 1862 raft_consensus.cc:515] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:18.513877 1744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "209968db4dee47c49497a66c3ee750b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "647c0de4ce394caca75179fe11658fc8"
I20251026 02:56:18.513938 1815 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "209968db4dee47c49497a66c3ee750b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751b719b75ce4bc2a88a19091230ecb7"
I20251026 02:56:18.514093 1744 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:18.514159 1815 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:18.517469 1744 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209968db4dee47c49497a66c3ee750b7 in term 1.
I20251026 02:56:18.517886 1815 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209968db4dee47c49497a66c3ee750b7 in term 1.
I20251026 02:56:18.518498 1647 leader_election.cc:304] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209968db4dee47c49497a66c3ee750b7, 751b719b75ce4bc2a88a19091230ecb7; no voters:
I20251026 02:56:18.520701 1862 leader_election.cc:290] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [CANDIDATE]: Term 1 election: Requested vote from peers 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655), 751b719b75ce4bc2a88a19091230ecb7 (127.1.150.60:33729)
I20251026 02:56:18.521176 1862 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:18.521624 1862 raft_consensus.cc:697] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 1 LEADER]: Becoming Leader. State: Replica: 209968db4dee47c49497a66c3ee750b7, State: Running, Role: LEADER
I20251026 02:56:18.522389 1862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:18.526631 1888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209968db4dee47c49497a66c3ee750b7. Latest consensus state: current_term: 1 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:18.527088 1888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: This master's current role is: LEADER
I20251026 02:56:18.528074 1890 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 02:56:18.530455 1890 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 02:56:18.539968 1744 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 1 FOLLOWER]: Refusing update from remote peer 209968db4dee47c49497a66c3ee750b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:18.539997 1815 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 1 FOLLOWER]: Refusing update from remote peer 209968db4dee47c49497a66c3ee750b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:18.541208 1862 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:18.541663 1888 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:18.565806 1848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209968db4dee47c49497a66c3ee750b7. Latest consensus state: current_term: 1 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:18.566134 1848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:18.570981 1876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209968db4dee47c49497a66c3ee750b7. Latest consensus state: current_term: 1 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:18.571244 1876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:18.576762 1862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:18.577070 1862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: This master's current role is: LEADER
I20251026 02:56:18.577368 1888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:18.577579 1888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: This master's current role is: LEADER
I20251026 02:56:18.579407 1893 mvcc.cc:204] Tried to move back new op lower bound from 7214888462488219648 to 7214888462434156544. Current Snapshot: MvccSnapshot[applied={T|T < 7214888462488219648}]
I20251026 02:56:18.583354 1848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:18.583578 1848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:18.587283 1876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:18.587493 1876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:18.587710 1890 catalog_manager.cc:1357] Generated new cluster ID: 98a29211b5ce4bf98616dd4b8985c732
I20251026 02:56:18.587843 1890 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 02:56:18.628252 1890 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 02:56:18.630071 1890 catalog_manager.cc:1514] Loading token signing keys...
I20251026 02:56:18.643311 1890 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Generated new TSK 0
I20251026 02:56:18.645232 1890 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 02:56:18.659927 1624 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
/tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.150.1:0
--local_ip_for_outbound_sockets=127.1.150.1
--webserver_interface=127.1.150.1
--webserver_port=0
--tserver_master_addrs=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--builtin_ntp_servers=127.1.150.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20251026 02:56:19.002316 1900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 02:56:19.002643 1900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 02:56:19.002779 1900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 02:56:19.011655 1900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 02:56:19.011878 1900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.150.1
I20251026 02:56:19.021837 1900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.150.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.150.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.150.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.150.1
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 26 Oct 2025 02:43:21 UTC on 24a791456cd2
build id 8720
ASAN enabled
I20251026 02:56:19.023492 1900 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 02:56:19.025354 1900 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 02:56:19.035241 1906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:19.035266 1905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:19.035876 1908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 02:56:19.036476 1900 server_base.cc:1047] running on GCE node
I20251026 02:56:19.037072 1900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 02:56:19.038070 1900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 02:56:19.039311 1900 hybrid_clock.cc:648] HybridClock initialized: now 1761447379039244 us; error 61 us; skew 500 ppm
I20251026 02:56:19.039644 1900 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 02:56:19.042024 1900 webserver.cc:492] Webserver started at http://127.1.150.1:38297/ using document root <none> and password file <none>
I20251026 02:56:19.042804 1900 fs_manager.cc:362] Metadata directory not provided
I20251026 02:56:19.042901 1900 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 02:56:19.043249 1900 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 02:56:19.045869 1900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/data/instance:
uuid: "7fef01f4f2b843538ea245a09dd087a8"
format_stamp: "Formatted at 2025-10-26 02:56:19 on dist-test-slave-h3w0"
I20251026 02:56:19.046727 1900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/instance:
uuid: "7fef01f4f2b843538ea245a09dd087a8"
format_stamp: "Formatted at 2025-10-26 02:56:19 on dist-test-slave-h3w0"
I20251026 02:56:19.052282 1900 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20251026 02:56:19.055747 1914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:19.057232 1900 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20251026 02:56:19.057399 1900 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal
uuid: "7fef01f4f2b843538ea245a09dd087a8"
format_stamp: "Formatted at 2025-10-26 02:56:19 on dist-test-slave-h3w0"
I20251026 02:56:19.057608 1900 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 02:56:19.080080 1900 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 02:56:19.080952 1900 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 02:56:19.081235 1900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 02:56:19.082237 1900 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 02:56:19.084029 1900 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 02:56:19.084127 1900 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:19.084261 1900 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 02:56:19.084340 1900 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:19.115521 1900 rpc_server.cc:307] RPC server started. Bound to: 127.1.150.1:37563
I20251026 02:56:19.115594 2026 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.150.1:37563 every 8 connection(s)
I20251026 02:56:19.116969 1900 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/data/info.pb
I20251026 02:56:19.117991 1624 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu as pid 1900
I20251026 02:56:19.118268 1624 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/instance
I20251026 02:56:19.136755 1624 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
/tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.150.2:0
--local_ip_for_outbound_sockets=127.1.150.2
--webserver_interface=127.1.150.2
--webserver_port=0
--tserver_master_addrs=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--builtin_ntp_servers=127.1.150.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251026 02:56:19.140698 2029 heartbeater.cc:344] Connected to a master server at 127.1.150.61:37655
I20251026 02:56:19.141808 2029 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:19.143026 2029 heartbeater.cc:507] Master 127.1.150.61:37655 requested a full tablet report, sending...
I20251026 02:56:19.143545 2028 heartbeater.cc:344] Connected to a master server at 127.1.150.62:44525
I20251026 02:56:19.143700 2028 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:19.144268 2028 heartbeater.cc:507] Master 127.1.150.62:44525 requested a full tablet report, sending...
I20251026 02:56:19.144788 2027 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:19.144903 2027 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:19.145406 2027 heartbeater.cc:507] Master 127.1.150.60:33729 requested a full tablet report, sending...
I20251026 02:56:19.145939 1662 ts_manager.cc:194] Registered new tserver with Master: 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:19.145920 1734 ts_manager.cc:194] Registered new tserver with Master: 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:19.147212 1805 ts_manager.cc:194] Registered new tserver with Master: 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:19.148314 1662 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.150.1:47931
I20251026 02:56:19.255515 1860 catalog_manager.cc:1269] Loaded cluster ID: 98a29211b5ce4bf98616dd4b8985c732
I20251026 02:56:19.255640 1860 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: loading cluster ID for follower catalog manager: success
I20251026 02:56:19.259528 1860 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: acquiring CA information for follower catalog manager: success
I20251026 02:56:19.261950 1860 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 02:56:19.461720 1887 catalog_manager.cc:1269] Loaded cluster ID: 98a29211b5ce4bf98616dd4b8985c732
I20251026 02:56:19.461874 1887 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8: loading cluster ID for follower catalog manager: success
I20251026 02:56:19.466149 1887 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8: acquiring CA information for follower catalog manager: success
I20251026 02:56:19.468927 1887 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251026 02:56:19.489975 2037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 02:56:19.490294 2037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 02:56:19.490433 2037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 02:56:19.498987 2037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 02:56:19.499205 2037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.150.2
I20251026 02:56:19.509231 2037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.150.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.150.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.150.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.150.2
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 26 Oct 2025 02:43:21 UTC on 24a791456cd2
build id 8720
ASAN enabled
I20251026 02:56:19.510876 2037 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 02:56:19.512674 2037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 02:56:19.521715 2044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:19.521710 2043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:19.523217 2046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 02:56:19.524032 2037 server_base.cc:1047] running on GCE node
I20251026 02:56:19.524639 2037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 02:56:19.525529 2037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 02:56:19.526733 2037 hybrid_clock.cc:648] HybridClock initialized: now 1761447379526668 us; error 58 us; skew 500 ppm
I20251026 02:56:19.527065 2037 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 02:56:19.529398 2037 webserver.cc:492] Webserver started at http://127.1.150.2:41863/ using document root <none> and password file <none>
I20251026 02:56:19.530176 2037 fs_manager.cc:362] Metadata directory not provided
I20251026 02:56:19.530282 2037 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 02:56:19.530628 2037 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 02:56:19.533087 2037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/data/instance:
uuid: "0b6b0a6c4e144982b67980118cddba48"
format_stamp: "Formatted at 2025-10-26 02:56:19 on dist-test-slave-h3w0"
I20251026 02:56:19.533977 2037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/instance:
uuid: "0b6b0a6c4e144982b67980118cddba48"
format_stamp: "Formatted at 2025-10-26 02:56:19 on dist-test-slave-h3w0"
I20251026 02:56:19.539878 2037 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20251026 02:56:19.543727 2052 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:19.545212 2037 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251026 02:56:19.545372 2037 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal
uuid: "0b6b0a6c4e144982b67980118cddba48"
format_stamp: "Formatted at 2025-10-26 02:56:19 on dist-test-slave-h3w0"
I20251026 02:56:19.545588 2037 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 02:56:19.571038 2037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 02:56:19.571918 2037 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 02:56:19.572253 2037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 02:56:19.573385 2037 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 02:56:19.575193 2037 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 02:56:19.575284 2037 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:19.575387 2037 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 02:56:19.575466 2037 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:19.611703 2037 rpc_server.cc:307] RPC server started. Bound to: 127.1.150.2:44611
I20251026 02:56:19.611747 2164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.150.2:44611 every 8 connection(s)
I20251026 02:56:19.613266 2037 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/data/info.pb
I20251026 02:56:19.620846 1624 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu as pid 2037
I20251026 02:56:19.621053 1624 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/instance
I20251026 02:56:19.626711 1624 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
/tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.150.3:0
--local_ip_for_outbound_sockets=127.1.150.3
--webserver_interface=127.1.150.3
--webserver_port=0
--tserver_master_addrs=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--builtin_ntp_servers=127.1.150.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251026 02:56:19.630002 2171 raft_consensus.cc:493] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 209968db4dee47c49497a66c3ee750b7)
I20251026 02:56:19.632357 2171 raft_consensus.cc:515] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:19.634677 1673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "647c0de4ce394caca75179fe11658fc8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "209968db4dee47c49497a66c3ee750b7" is_pre_election: true
I20251026 02:56:19.636104 2167 heartbeater.cc:344] Connected to a master server at 127.1.150.61:37655
I20251026 02:56:19.636647 2167 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:19.637470 2171 leader_election.cc:290] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 209968db4dee47c49497a66c3ee750b7 (127.1.150.62:44525), 751b719b75ce4bc2a88a19091230ecb7 (127.1.150.60:33729)
I20251026 02:56:19.637627 1815 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "647c0de4ce394caca75179fe11658fc8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "751b719b75ce4bc2a88a19091230ecb7" is_pre_election: true
I20251026 02:56:19.638113 2167 heartbeater.cc:507] Master 127.1.150.61:37655 requested a full tablet report, sending...
I20251026 02:56:19.638545 1718 leader_election.cc:304] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 647c0de4ce394caca75179fe11658fc8; no voters: 209968db4dee47c49497a66c3ee750b7, 751b719b75ce4bc2a88a19091230ecb7
I20251026 02:56:19.639794 2171 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:19.640286 2166 heartbeater.cc:344] Connected to a master server at 127.1.150.62:44525
I20251026 02:56:19.640405 2166 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:19.640446 1734 ts_manager.cc:194] Registered new tserver with Master: 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:19.640967 2166 heartbeater.cc:507] Master 127.1.150.62:44525 requested a full tablet report, sending...
I20251026 02:56:19.642081 1662 ts_manager.cc:194] Registered new tserver with Master: 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:19.643185 1662 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.150.2:34337
I20251026 02:56:19.646944 2174 raft_consensus.cc:493] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 209968db4dee47c49497a66c3ee750b7)
I20251026 02:56:19.647203 2174 raft_consensus.cc:515] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:19.649626 1673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "209968db4dee47c49497a66c3ee750b7" is_pre_election: true
I20251026 02:56:19.651209 1743 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "647c0de4ce394caca75179fe11658fc8" is_pre_election: true
I20251026 02:56:19.651340 2028 heartbeater.cc:499] Master 127.1.150.62:44525 was elected leader, sending a full tablet report...
I20251026 02:56:19.651428 1743 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 751b719b75ce4bc2a88a19091230ecb7 in term 1.
I20251026 02:56:19.652421 1792 leader_election.cc:304] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 647c0de4ce394caca75179fe11658fc8, 751b719b75ce4bc2a88a19091230ecb7; no voters: 209968db4dee47c49497a66c3ee750b7
I20251026 02:56:19.653575 2175 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 02:56:19.653731 2175 raft_consensus.cc:493] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 209968db4dee47c49497a66c3ee750b7)
I20251026 02:56:19.653957 2165 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:19.653965 2175 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:19.654138 2165 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:19.654047 2174 leader_election.cc:290] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 209968db4dee47c49497a66c3ee750b7 (127.1.150.62:44525), 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655)
I20251026 02:56:19.654610 2165 heartbeater.cc:507] Master 127.1.150.60:33729 requested a full tablet report, sending...
I20251026 02:56:19.657377 2175 raft_consensus.cc:515] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:19.658169 1805 ts_manager.cc:194] Registered new tserver with Master: 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:19.658763 2175 leader_election.cc:290] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 2 election: Requested vote from peers 209968db4dee47c49497a66c3ee750b7 (127.1.150.62:44525), 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655)
I20251026 02:56:19.659150 1673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "209968db4dee47c49497a66c3ee750b7"
I20251026 02:56:19.659504 1743 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "647c0de4ce394caca75179fe11658fc8"
I20251026 02:56:19.659684 1743 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:19.662886 1743 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751b719b75ce4bc2a88a19091230ecb7 in term 2.
I20251026 02:56:19.663609 1792 leader_election.cc:304] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 647c0de4ce394caca75179fe11658fc8, 751b719b75ce4bc2a88a19091230ecb7; no voters: 209968db4dee47c49497a66c3ee750b7
I20251026 02:56:19.664237 2175 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 02:56:19.664525 1743 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 2 FOLLOWER]: Rejecting Update request from peer 209968db4dee47c49497a66c3ee750b7 for earlier term 1. Current term is 2. Ops: []
I20251026 02:56:19.664577 2175 raft_consensus.cc:697] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 2 LEADER]: Becoming Leader. State: Replica: 751b719b75ce4bc2a88a19091230ecb7, State: Running, Role: LEADER
I20251026 02:56:19.665081 2175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:19.665602 2176 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 02:56:19.679533 2177 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 02:56:19.679873 2177 raft_consensus.cc:740] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 209968db4dee47c49497a66c3ee750b7, State: Running, Role: LEADER
I20251026 02:56:19.680335 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 751b719b75ce4bc2a88a19091230ecb7. Latest consensus state: current_term: 2 leader_uuid: "751b719b75ce4bc2a88a19091230ecb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:19.680615 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: This master's current role is: LEADER
I20251026 02:56:19.680470 2177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:19.681478 2177 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:19.686492 2179 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 02:56:19.688277 2179 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 02:56:19.689075 2179 catalog_manager.cc:1269] Loaded cluster ID: 98a29211b5ce4bf98616dd4b8985c732
I20251026 02:56:19.689178 2179 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 02:56:19.690307 2179 catalog_manager.cc:1514] Loading token signing keys...
I20251026 02:56:19.691100 2179 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Loaded TSK: 0
I20251026 02:56:19.691890 2179 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251026 02:56:20.005441 2173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 02:56:20.005755 2173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 02:56:20.005892 2173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 02:56:20.013909 2173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 02:56:20.014113 2173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.150.3
I20251026 02:56:20.023816 2173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.150.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.150.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.150.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.150.3
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 26 Oct 2025 02:43:21 UTC on 24a791456cd2
build id 8720
ASAN enabled
I20251026 02:56:20.025535 2173 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 02:56:20.027372 2173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 02:56:20.037273 2185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:20.037222 2184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:20.037240 2187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 02:56:20.038419 2173 server_base.cc:1047] running on GCE node
I20251026 02:56:20.039042 2173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 02:56:20.040056 2173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 02:56:20.041275 2173 hybrid_clock.cc:648] HybridClock initialized: now 1761447380041219 us; error 53 us; skew 500 ppm
I20251026 02:56:20.041607 2173 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 02:56:20.043869 2173 webserver.cc:492] Webserver started at http://127.1.150.3:33345/ using document root <none> and password file <none>
I20251026 02:56:20.044672 2173 fs_manager.cc:362] Metadata directory not provided
I20251026 02:56:20.044780 2173 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 02:56:20.045158 2173 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 02:56:20.047744 2173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/data/instance:
uuid: "696476265495492790739ec126a08539"
format_stamp: "Formatted at 2025-10-26 02:56:20 on dist-test-slave-h3w0"
I20251026 02:56:20.048553 2173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/instance:
uuid: "696476265495492790739ec126a08539"
format_stamp: "Formatted at 2025-10-26 02:56:20 on dist-test-slave-h3w0"
I20251026 02:56:20.054000 2173 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20251026 02:56:20.057376 2193 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:20.058705 2173 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251026 02:56:20.058889 2173 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal
uuid: "696476265495492790739ec126a08539"
format_stamp: "Formatted at 2025-10-26 02:56:20 on dist-test-slave-h3w0"
I20251026 02:56:20.059084 2173 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 02:56:20.072324 2177 raft_consensus.cc:493] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:20.072527 2177 raft_consensus.cc:515] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:20.073338 2177 leader_election.cc:290] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655), 751b719b75ce4bc2a88a19091230ecb7 (127.1.150.60:33729)
I20251026 02:56:20.073855 1815 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "209968db4dee47c49497a66c3ee750b7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "751b719b75ce4bc2a88a19091230ecb7" is_pre_election: true
I20251026 02:56:20.073921 1743 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "209968db4dee47c49497a66c3ee750b7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "647c0de4ce394caca75179fe11658fc8" is_pre_election: true
I20251026 02:56:20.074133 1743 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209968db4dee47c49497a66c3ee750b7 in term 2.
I20251026 02:56:20.074527 1650 leader_election.cc:304] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 209968db4dee47c49497a66c3ee750b7, 647c0de4ce394caca75179fe11658fc8; no voters: 751b719b75ce4bc2a88a19091230ecb7
I20251026 02:56:20.074815 2177 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 02:56:20.074932 2177 raft_consensus.cc:493] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:20.075017 2177 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 2 FOLLOWER]: Advancing to term 3
I20251026 02:56:20.077217 2177 raft_consensus.cc:515] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:20.077829 2177 leader_election.cc:290] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [CANDIDATE]: Term 3 election: Requested vote from peers 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655), 751b719b75ce4bc2a88a19091230ecb7 (127.1.150.60:33729)
I20251026 02:56:20.078329 1743 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "209968db4dee47c49497a66c3ee750b7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "647c0de4ce394caca75179fe11658fc8"
I20251026 02:56:20.078403 1815 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "209968db4dee47c49497a66c3ee750b7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "751b719b75ce4bc2a88a19091230ecb7"
I20251026 02:56:20.078557 1743 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 2 FOLLOWER]: Advancing to term 3
I20251026 02:56:20.080926 1743 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209968db4dee47c49497a66c3ee750b7 in term 3.
I20251026 02:56:20.081398 1650 leader_election.cc:304] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 209968db4dee47c49497a66c3ee750b7, 647c0de4ce394caca75179fe11658fc8; no voters: 751b719b75ce4bc2a88a19091230ecb7
I20251026 02:56:20.081733 2177 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 02:56:20.081889 2177 raft_consensus.cc:697] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 3 LEADER]: Becoming Leader. State: Replica: 209968db4dee47c49497a66c3ee750b7, State: Running, Role: LEADER
I20251026 02:56:20.082163 2177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:20.083469 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209968db4dee47c49497a66c3ee750b7. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:20.083748 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: This master's current role is: LEADER
I20251026 02:56:20.084355 2199 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 02:56:20.086156 2199 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 02:56:20.087116 2199 catalog_manager.cc:1269] Loaded cluster ID: 98a29211b5ce4bf98616dd4b8985c732
I20251026 02:56:20.087188 2199 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 02:56:20.088282 2173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 02:56:20.088389 2199 catalog_manager.cc:1514] Loading token signing keys...
I20251026 02:56:20.089226 2173 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 02:56:20.089241 2199 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Loaded TSK: 0
I20251026 02:56:20.089483 2173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 02:56:20.089934 2199 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 02:56:20.090586 2173 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 02:56:20.092366 2173 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 02:56:20.092474 2173 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:20.092592 2173 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 02:56:20.092669 2173 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:20.129168 2173 rpc_server.cc:307] RPC server started. Bound to: 127.1.150.3:38107
I20251026 02:56:20.129231 2307 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.150.3:38107 every 8 connection(s)
I20251026 02:56:20.130618 2173 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/data/info.pb
I20251026 02:56:20.138088 1624 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu as pid 2173
I20251026 02:56:20.138322 1624 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/instance
I20251026 02:56:20.146700 2166 heartbeater.cc:499] Master 127.1.150.62:44525 was elected leader, sending a full tablet report...
I20251026 02:56:20.149405 2310 heartbeater.cc:344] Connected to a master server at 127.1.150.61:37655
I20251026 02:56:20.149493 2308 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:20.149840 2310 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:20.149844 2308 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:20.149956 2309 heartbeater.cc:344] Connected to a master server at 127.1.150.62:44525
I20251026 02:56:20.150089 2309 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:20.151080 2309 heartbeater.cc:507] Master 127.1.150.62:44525 requested a full tablet report, sending...
I20251026 02:56:20.151299 2308 heartbeater.cc:507] Master 127.1.150.60:33729 requested a full tablet report, sending...
I20251026 02:56:20.152163 2310 heartbeater.cc:507] Master 127.1.150.61:37655 requested a full tablet report, sending...
I20251026 02:56:20.152881 1663 ts_manager.cc:194] Registered new tserver with Master: 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:20.152932 1805 ts_manager.cc:194] Registered new tserver with Master: 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:20.153177 1733 ts_manager.cc:194] Registered new tserver with Master: 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:20.154300 1663 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.150.3:36919
I20251026 02:56:20.154888 1805 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.150.3:52067
I20251026 02:56:20.166622 1624 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 02:56:20.196676 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2f66e450af1b48f7a7cecb4bd3714df8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:20.197310 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_92019775b27a49c59fbab5a7f5ceee18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:20.197669 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_73998ab2e381438781f0c24195d78d60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.198518 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92019775b27a49c59fbab5a7f5ceee18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.198833 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d844ca1ecad04041ad6baad492f1b63c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.199460 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d844ca1ecad04041ad6baad492f1b63c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:20.198534 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f66e450af1b48f7a7cecb4bd3714df8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:20.198544 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73998ab2e381438781f0c24195d78d60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.201864 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f070d68ce8e342ffb918288c7b46f3f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.202559 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f070d68ce8e342ffb918288c7b46f3f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.206848 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0d997fa5825e4cf2a797881f912b609f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:20.207669 1743 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 3 FOLLOWER]: Rejecting Update request from peer 751b719b75ce4bc2a88a19091230ecb7 for earlier term 2. Current term is 3. Ops: []
I20251026 02:56:20.208256 1673 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [term 3 LEADER]: Rejecting Update request from peer 751b719b75ce4bc2a88a19091230ecb7 for earlier term 2. Current term is 3. Ops: []
W20251026 02:56:20.208271 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d997fa5825e4cf2a797881f912b609f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.209458 2328 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 02:56:20.211349 1743 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [term 3 FOLLOWER]: Refusing update from remote peer 209968db4dee47c49497a66c3ee750b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251026 02:56:20.211989 2177 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 02:56:20.212543 1815 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 02:56:20.212674 1815 raft_consensus.cc:740] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 751b719b75ce4bc2a88a19091230ecb7, State: Running, Role: LEADER
I20251026 02:56:20.212972 1815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:20.213245 2332 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 02:56:20.213526 2332 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [NON_LEADER]: Peer 209968db4dee47c49497a66c3ee750b7 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251026 02:56:20.214404 1815 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 2 FOLLOWER]: Advancing to term 3
I20251026 02:56:20.215294 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8827f142daf64c45b7786c00d72ace16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.216279 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8827f142daf64c45b7786c00d72ace16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.215569 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1e76461bd0074b81bc88b51dc0f0af41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.217386 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e76461bd0074b81bc88b51dc0f0af41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.217746 1815 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Refusing update from remote peer 209968db4dee47c49497a66c3ee750b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20251026 02:56:20.218619 2333 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20251026 02:56:20.218667 2177 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 02:56:20.218842 2177 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [LEADER]: Peer 751b719b75ce4bc2a88a19091230ecb7 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251026 02:56:20.220698 1815 pending_rounds.cc:85] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Aborting all ops after (but not including) 4
I20251026 02:56:20.220871 1815 pending_rounds.cc:107] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20251026 02:56:20.220993 1815 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 02:56:20.222235 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ee829a4e2a6441949c34fcb073a6f34a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.223049 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee829a4e2a6441949c34fcb073a6f34a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.232980 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_bdb3447e91a34a10902f73730dbd953a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.233795 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdb3447e91a34a10902f73730dbd953a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.242529 2333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209968db4dee47c49497a66c3ee750b7. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:20.242858 2333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:20.245232 2329 mvcc.cc:204] Tried to move back new op lower bound from 7214888469340299264 to 7214888468818767872. Current Snapshot: MvccSnapshot[applied={T|T < 7214888469340299264}]
I20251026 02:56:20.265746 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2f66e450af1b48f7a7cecb4bd3714df8" } new_table_name: "default.Table_e4d2256ca9d340b78fe5fb50644c789a" modify_external_catalogs: true
I20251026 02:56:20.267907 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_92019775b27a49c59fbab5a7f5ceee18" } new_table_name: "default.Table_730bdb7afb23443fa04b12ea56c40fd6" modify_external_catalogs: true
I20251026 02:56:20.270172 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_73998ab2e381438781f0c24195d78d60" } new_table_name: "default.Table_7c60ae6cebc04434975a2c3782f4aaa7" modify_external_catalogs: true
I20251026 02:56:20.274838 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:20.275111 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: This master's current role is: LEADER
I20251026 02:56:20.275744 2333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:20.275987 2333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:20.276695 2335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209968db4dee47c49497a66c3ee750b7. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:20.276978 2335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:20.277033 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d844ca1ecad04041ad6baad492f1b63c" } new_table_name: "default.Table_448f8c6a1d0c4125af909cf6570e41ae" modify_external_catalogs: true
I20251026 02:56:20.279937 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:20.280254 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: This master's current role is: LEADER
I20251026 02:56:20.303140 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_bdb3447e91a34a10902f73730dbd953a" } new_table_name: "default.Table_77c64bf7bba64a87a15bd6df5de5521e" modify_external_catalogs: true
I20251026 02:56:20.306897 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ee829a4e2a6441949c34fcb073a6f34a" } modify_external_catalogs: true
I20251026 02:56:20.307266 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8827f142daf64c45b7786c00d72ace16" } modify_external_catalogs: true
I20251026 02:56:20.307644 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ee829a4e2a6441949c34fcb073a6f34a" } modify_external_catalogs: true
I20251026 02:56:20.309276 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f070d68ce8e342ffb918288c7b46f3f8" } modify_external_catalogs: true
I20251026 02:56:20.309450 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f070d68ce8e342ffb918288c7b46f3f8" } modify_external_catalogs: true
I20251026 02:56:20.310952 2335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:20.311156 2335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647c0de4ce394caca75179fe11658fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:20.311614 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0d997fa5825e4cf2a797881f912b609f" } modify_external_catalogs: true
I20251026 02:56:20.311866 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8827f142daf64c45b7786c00d72ace16" } modify_external_catalogs: true
I20251026 02:56:20.315061 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1e76461bd0074b81bc88b51dc0f0af41" } modify_external_catalogs: true
I20251026 02:56:20.315284 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1e76461bd0074b81bc88b51dc0f0af41" } modify_external_catalogs: true
I20251026 02:56:20.322196 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0d997fa5825e4cf2a797881f912b609f" } modify_external_catalogs: true
W20251026 02:56:20.326886 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 604b7272209d4652989c3e6d8201a255 has no consensus state
W20251026 02:56:20.328186 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f4436f64997445aa880610c5a93f955 has no consensus state
W20251026 02:56:20.328912 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 652112d5039943628550188522bf0a17 has no consensus state
W20251026 02:56:20.330938 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a4ba66147c249728713a42d20f541f4 consensus state has no leader
W20251026 02:56:20.331595 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d066a461d9e4df198968c2e6f11ae2d has no consensus state
I20251026 02:56:20.332803 1655 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 1317f50f2dbc4d09a17646cf20f2c136
I20251026 02:56:20.333925 1662 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 225cccf93e694de88bb34cb6dbc048dd
I20251026 02:56:20.357061 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ac43df5558634550a12aa65c521d8a14
W20251026 02:56:20.357187 1649 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 604b7272209d4652989c3e6d8201a255 (table default.Table_e4d2256ca9d340b78fe5fb50644c789a [id=d7ac91b3b98441baaa6ecc48cb904c7e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 604b7272209d4652989c3e6d8201a255 consensus state has no leader
I20251026 02:56:20.357511 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0b68c0b860984df99a7eeb73854a5da8
I20251026 02:56:20.358719 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet deb5e70eb5f14e2ab260691ce79176de
I20251026 02:56:20.391093 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet deb5e70eb5f14e2ab260691ce79176de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:20.391567 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b68c0b860984df99a7eeb73854a5da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:20.392706 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet ac43df5558634550a12aa65c521d8a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:20.393306 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 3a4ba66147c249728713a42d20f541f4 (DEFAULT_TABLE table=default.Table_448f8c6a1d0c4125af909cf6570e41ae [id=9e895e2eb27749839df30c205507d222]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.394294 2098 tablet_service.cc:1505] Processing CreateTablet for tablet deb5e70eb5f14e2ab260691ce79176de (DEFAULT_TABLE table=default.Table_f070d68ce8e342ffb918288c7b46f3f8 [id=c1d3e3f0c2b7403387d77c9efcde8e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.394877 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4ba66147c249728713a42d20f541f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.395054 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 0d066a461d9e4df198968c2e6f11ae2d (DEFAULT_TABLE table=default.Table_77c64bf7bba64a87a15bd6df5de5521e [id=3d118d55d42047bdbc0b8fb944b3a190]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:20.395117 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet deb5e70eb5f14e2ab260691ce79176de because the tablet was not found. No further retry: Not found: Tablet not found: deb5e70eb5f14e2ab260691ce79176de
I20251026 02:56:20.395514 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d066a461d9e4df198968c2e6f11ae2d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:20.395925 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0b68c0b860984df99a7eeb73854a5da8 because the tablet was not found. No further retry: Not found: Tablet not found: 0b68c0b860984df99a7eeb73854a5da8
I20251026 02:56:20.396347 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 0b68c0b860984df99a7eeb73854a5da8 (DEFAULT_TABLE table=default.Table_0d997fa5825e4cf2a797881f912b609f [id=319c8d881ba442c6a0629a979e01b289]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:20.396554 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ac43df5558634550a12aa65c521d8a14 because the tablet was not found. No further retry: Not found: Tablet not found: ac43df5558634550a12aa65c521d8a14
I20251026 02:56:20.396814 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b68c0b860984df99a7eeb73854a5da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.397347 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 652112d5039943628550188522bf0a17 (DEFAULT_TABLE table=default.Table_730bdb7afb23443fa04b12ea56c40fd6 [id=01000ef9384644bdb76c6cba086b605d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.397799 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652112d5039943628550188522bf0a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.399186 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 3f4436f64997445aa880610c5a93f955 (DEFAULT_TABLE table=default.Table_7c60ae6cebc04434975a2c3782f4aaa7 [id=9d77f9be68084772a1f3ee4d00a5180d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.399778 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4436f64997445aa880610c5a93f955. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.400246 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deb5e70eb5f14e2ab260691ce79176de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.404291 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 3a4ba66147c249728713a42d20f541f4 (DEFAULT_TABLE table=default.Table_448f8c6a1d0c4125af909cf6570e41ae [id=9e895e2eb27749839df30c205507d222]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.405822 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4ba66147c249728713a42d20f541f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.413275 1960 tablet_service.cc:1505] Processing CreateTablet for tablet deb5e70eb5f14e2ab260691ce79176de (DEFAULT_TABLE table=default.Table_f070d68ce8e342ffb918288c7b46f3f8 [id=c1d3e3f0c2b7403387d77c9efcde8e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.413421 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 652112d5039943628550188522bf0a17 (DEFAULT_TABLE table=default.Table_730bdb7afb23443fa04b12ea56c40fd6 [id=01000ef9384644bdb76c6cba086b605d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.414028 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deb5e70eb5f14e2ab260691ce79176de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.414768 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 3f4436f64997445aa880610c5a93f955 (DEFAULT_TABLE table=default.Table_7c60ae6cebc04434975a2c3782f4aaa7 [id=9d77f9be68084772a1f3ee4d00a5180d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.415995 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b68c0b860984df99a7eeb73854a5da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.416709 1958 tablet_service.cc:1505] Processing CreateTablet for tablet ac43df5558634550a12aa65c521d8a14 (DEFAULT_TABLE table=default.Table_8827f142daf64c45b7786c00d72ace16 [id=c881f3e0b92f45788e52269792793f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.417169 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet ac43df5558634550a12aa65c521d8a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.418438 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4436f64997445aa880610c5a93f955. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.420761 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac43df5558634550a12aa65c521d8a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.423173 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 0d066a461d9e4df198968c2e6f11ae2d (DEFAULT_TABLE table=default.Table_77c64bf7bba64a87a15bd6df5de5521e [id=3d118d55d42047bdbc0b8fb944b3a190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.423827 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d066a461d9e4df198968c2e6f11ae2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.424789 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 0b68c0b860984df99a7eeb73854a5da8 (DEFAULT_TABLE table=default.Table_0d997fa5825e4cf2a797881f912b609f [id=319c8d881ba442c6a0629a979e01b289]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.425477 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b68c0b860984df99a7eeb73854a5da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.428498 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet deb5e70eb5f14e2ab260691ce79176de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.429199 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652112d5039943628550188522bf0a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.393240 2096 tablet_service.cc:1505] Processing CreateTablet for tablet ac43df5558634550a12aa65c521d8a14 (DEFAULT_TABLE table=default.Table_8827f142daf64c45b7786c00d72ace16 [id=c881f3e0b92f45788e52269792793f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.432937 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac43df5558634550a12aa65c521d8a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.404417 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 604b7272209d4652989c3e6d8201a255 (DEFAULT_TABLE table=default.Table_e4d2256ca9d340b78fe5fb50644c789a [id=d7ac91b3b98441baaa6ecc48cb904c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.440867 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 604b7272209d4652989c3e6d8201a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.442926 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 604b7272209d4652989c3e6d8201a255 (DEFAULT_TABLE table=default.Table_e4d2256ca9d340b78fe5fb50644c789a [id=d7ac91b3b98441baaa6ecc48cb904c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.444994 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 604b7272209d4652989c3e6d8201a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.449471 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 3a4ba66147c249728713a42d20f541f4 (DEFAULT_TABLE table=default.Table_448f8c6a1d0c4125af909cf6570e41ae [id=9e895e2eb27749839df30c205507d222]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.450091 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4ba66147c249728713a42d20f541f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.449999 2241 tablet_service.cc:1505] Processing CreateTablet for tablet deb5e70eb5f14e2ab260691ce79176de (DEFAULT_TABLE table=default.Table_f070d68ce8e342ffb918288c7b46f3f8 [id=c1d3e3f0c2b7403387d77c9efcde8e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.450673 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deb5e70eb5f14e2ab260691ce79176de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.451985 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 652112d5039943628550188522bf0a17 (DEFAULT_TABLE table=default.Table_730bdb7afb23443fa04b12ea56c40fd6 [id=01000ef9384644bdb76c6cba086b605d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.452364 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 604b7272209d4652989c3e6d8201a255 (DEFAULT_TABLE table=default.Table_e4d2256ca9d340b78fe5fb50644c789a [id=d7ac91b3b98441baaa6ecc48cb904c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.452539 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652112d5039943628550188522bf0a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.452500 2239 tablet_service.cc:1505] Processing CreateTablet for tablet ac43df5558634550a12aa65c521d8a14 (DEFAULT_TABLE table=default.Table_8827f142daf64c45b7786c00d72ace16 [id=c881f3e0b92f45788e52269792793f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.452909 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 604b7272209d4652989c3e6d8201a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.453109 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac43df5558634550a12aa65c521d8a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.455996 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 3f4436f64997445aa880610c5a93f955 (DEFAULT_TABLE table=default.Table_7c60ae6cebc04434975a2c3782f4aaa7 [id=9d77f9be68084772a1f3ee4d00a5180d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.456645 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4436f64997445aa880610c5a93f955. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.458777 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 0b68c0b860984df99a7eeb73854a5da8 (DEFAULT_TABLE table=default.Table_0d997fa5825e4cf2a797881f912b609f [id=319c8d881ba442c6a0629a979e01b289]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.459159 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 0d066a461d9e4df198968c2e6f11ae2d (DEFAULT_TABLE table=default.Table_77c64bf7bba64a87a15bd6df5de5521e [id=3d118d55d42047bdbc0b8fb944b3a190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.459358 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b68c0b860984df99a7eeb73854a5da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.459694 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d066a461d9e4df198968c2e6f11ae2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.455437 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet deb5e70eb5f14e2ab260691ce79176de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.454879 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet ac43df5558634550a12aa65c521d8a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.454694 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b68c0b860984df99a7eeb73854a5da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.473519 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f0f39c08cd4d4ae28523c4fd14828ef2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.474406 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0f39c08cd4d4ae28523c4fd14828ef2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.477339 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c6bad2c857bb4c9fb8f8e2f5a87344cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:20.477741 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d15145ea85264fc9aab94645c7ff3b41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.477955 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6bad2c857bb4c9fb8f8e2f5a87344cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:20.478350 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d15145ea85264fc9aab94645c7ff3b41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.479233 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a5da05aa982c45c58da3b2ad4697f283"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:20.479487 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_99b42c4a4a874b68bb4beba4c08cdeb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.479658 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5da05aa982c45c58da3b2ad4697f283 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:20.479979 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99b42c4a4a874b68bb4beba4c08cdeb0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.477393 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9205842d965c49bfae1d1511812f0caf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.480587 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9205842d965c49bfae1d1511812f0caf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.481495 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_528abc1c5aa443a89795cc31b809be9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.481938 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_528abc1c5aa443a89795cc31b809be9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.486526 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1b8997fc8d6b4f788983c76e84b2d0c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.487141 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b8997fc8d6b4f788983c76e84b2d0c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.493713 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0aa77747f2be46d8be9e7c5873940ca1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.494362 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0aa77747f2be46d8be9e7c5873940ca1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.495934 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_664f6e878825452ebaeaed2a7a0aefc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.496556 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_664f6e878825452ebaeaed2a7a0aefc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.528429 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet deb5e70eb5f14e2ab260691ce79176de from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:20.529161 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 3a4ba66147c249728713a42d20f541f4 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:20.536092 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_730bdb7afb23443fa04b12ea56c40fd6" } new_table_name: "default.Table_233cc7c9cac543889a30f7a5a8d22156" modify_external_catalogs: true
I20251026 02:56:20.536749 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9205842d965c49bfae1d1511812f0caf" } new_table_name: "default.Table_cfef4f9714bd4b77b3a9c61ccbc25684" modify_external_catalogs: true
I20251026 02:56:20.537923 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet deb5e70eb5f14e2ab260691ce79176de from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:20.538616 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_77c64bf7bba64a87a15bd6df5de5521e" } new_table_name: "default.Table_a445cc1241b548788d59b8dc29fa2f2b" modify_external_catalogs: true
I20251026 02:56:20.543607 2375 tablet_bootstrap.cc:492] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:20.545737 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0b68c0b860984df99a7eeb73854a5da8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b68c0b860984df99a7eeb73854a5da8 already in progress: creating tablet
W20251026 02:56:20.546181 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ac43df5558634550a12aa65c521d8a14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac43df5558634550a12aa65c521d8a14 already in progress: creating tablet
W20251026 02:56:20.546504 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet deb5e70eb5f14e2ab260691ce79176de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deb5e70eb5f14e2ab260691ce79176de already in progress: creating tablet
W20251026 02:56:20.546782 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet deb5e70eb5f14e2ab260691ce79176de because the tablet was not found. No further retry: Not found: Tablet not found: deb5e70eb5f14e2ab260691ce79176de
W20251026 02:56:20.547053 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ac43df5558634550a12aa65c521d8a14 because the tablet was not found. No further retry: Not found: Tablet not found: ac43df5558634550a12aa65c521d8a14
W20251026 02:56:20.547319 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0b68c0b860984df99a7eeb73854a5da8 because the tablet was not found. No further retry: Not found: Tablet not found: 0b68c0b860984df99a7eeb73854a5da8
I20251026 02:56:20.552676 1955 tablet_service.cc:1505] Processing CreateTablet for tablet de24439a6a3a4734a08160cb7393429a (DEFAULT_TABLE table=default.Table_f0f39c08cd4d4ae28523c4fd14828ef2 [id=931430a4a03741bd86175eabf9764a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.552865 2100 tablet_service.cc:1505] Processing CreateTablet for tablet de24439a6a3a4734a08160cb7393429a (DEFAULT_TABLE table=default.Table_f0f39c08cd4d4ae28523c4fd14828ef2 [id=931430a4a03741bd86175eabf9764a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.553267 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de24439a6a3a4734a08160cb7393429a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.553308 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de24439a6a3a4734a08160cb7393429a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.557001 2237 tablet_service.cc:1505] Processing CreateTablet for tablet de24439a6a3a4734a08160cb7393429a (DEFAULT_TABLE table=default.Table_f0f39c08cd4d4ae28523c4fd14828ef2 [id=931430a4a03741bd86175eabf9764a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.557516 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de24439a6a3a4734a08160cb7393429a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.558411 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f0f39c08cd4d4ae28523c4fd14828ef2" } new_table_name: "default.Table_3c222bf9316b4ffe89d6899f4ded8349" modify_external_catalogs: true
I20251026 02:56:20.560003 2375 tablet_bootstrap.cc:654] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:20.560621 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a5da05aa982c45c58da3b2ad4697f283" } new_table_name: "default.Table_6175fc5b75b74fe5a7abe8861a2df365" modify_external_catalogs: true
I20251026 02:56:20.561565 2375 log.cc:826] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48: Log is configured to *not* fsync() on all Append() calls
I20251026 02:56:20.567081 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet 652112d5039943628550188522bf0a17 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:20.571468 2381 tablet_bootstrap.cc:492] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:20.571774 2378 tablet_bootstrap.cc:492] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:20.578075 2378 tablet_bootstrap.cc:654] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:20.578897 2375 tablet_bootstrap.cc:492] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:20.579226 2375 ts_tablet_manager.cc:1403] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.037s user 0.014s sys 0.000s
I20251026 02:56:20.579792 2381 tablet_bootstrap.cc:654] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:20.579842 2378 log.cc:826] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8: Log is configured to *not* fsync() on all Append() calls
I20251026 02:56:20.581387 2381 log.cc:826] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539: Log is configured to *not* fsync() on all Append() calls
W20251026 02:56:20.581588 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 652112d5039943628550188522bf0a17 consensus state has no leader
W20251026 02:56:20.582307 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7bc92f90a68b4e749bfd4fa49076bfe2 has no consensus state
I20251026 02:56:20.583163 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0aa77747f2be46d8be9e7c5873940ca1" } modify_external_catalogs: true
I20251026 02:56:20.583379 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0aa77747f2be46d8be9e7c5873940ca1" } modify_external_catalogs: true
W20251026 02:56:20.584558 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d066a461d9e4df198968c2e6f11ae2d consensus state has no leader
I20251026 02:56:20.586661 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3a4ba66147c249728713a42d20f541f4
I20251026 02:56:20.587203 1655 catalog_manager.cc:6719] ReplaceTablet: tablet 3a4ba66147c249728713a42d20f541f4 deleted and replaced by tablet 630beab92a0d4b928c79d9f0c32795f0
W20251026 02:56:20.588068 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de24439a6a3a4734a08160cb7393429a consensus state has no leader
I20251026 02:56:20.588299 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4ba66147c249728713a42d20f541f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 630beab92a0d4b928c79d9f0c32795f0) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.588642 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4ba66147c249728713a42d20f541f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 630beab92a0d4b928c79d9f0c32795f0) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:20.588766 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82315e4c5a554eeca5048a496498ca1f has no consensus state
I20251026 02:56:20.588968 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4ba66147c249728713a42d20f541f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 630beab92a0d4b928c79d9f0c32795f0) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.589162 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 652112d5039943628550188522bf0a17
I20251026 02:56:20.589587 1662 catalog_manager.cc:6719] ReplaceTablet: tablet 652112d5039943628550188522bf0a17 deleted and replaced by tablet 701f13f90f154e378b353e84d1bebd4e
W20251026 02:56:20.591205 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3a4ba66147c249728713a42d20f541f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a4ba66147c249728713a42d20f541f4 already in progress: creating tablet
I20251026 02:56:20.591702 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 652112d5039943628550188522bf0a17 with delete_type TABLET_DATA_DELETED (replaced by tablet 701f13f90f154e378b353e84d1bebd4e) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:20.591638 2375 raft_consensus.cc:359] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.592113 2375 raft_consensus.cc:385] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:20.592239 2375 raft_consensus.cc:740] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
W20251026 02:56:20.593066 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 652112d5039943628550188522bf0a17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 652112d5039943628550188522bf0a17 already in progress: creating tablet
I20251026 02:56:20.592980 2375 consensus_queue.cc:260] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.594873 2381 tablet_bootstrap.cc:492] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:20.595219 2381 ts_tablet_manager.cc:1403] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20251026 02:56:20.597612 2378 tablet_bootstrap.cc:492] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:20.597983 2378 ts_tablet_manager.cc:1403] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.009s
I20251026 02:56:20.607417 2378 raft_consensus.cc:359] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.608335 2378 raft_consensus.cc:385] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:20.608508 2378 raft_consensus.cc:740] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:20.610256 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 20c39a62bfab429bb231ae073585d0e5 (DEFAULT_TABLE table=default.Table_0aa77747f2be46d8be9e7c5873940ca1 [id=a414f870eb244e0ba5d2569c6c91edc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.610170 2381 raft_consensus.cc:359] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.610577 2381 raft_consensus.cc:385] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:20.610668 2381 raft_consensus.cc:740] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:20.610729 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c39a62bfab429bb231ae073585d0e5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:20.615657 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3a4ba66147c249728713a42d20f541f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a4ba66147c249728713a42d20f541f4 already in progress: creating tablet
W20251026 02:56:20.615958 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3a4ba66147c249728713a42d20f541f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a4ba66147c249728713a42d20f541f4 already in progress: creating tablet
I20251026 02:56:20.616374 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet ac43df5558634550a12aa65c521d8a14 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:20.616587 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 82315e4c5a554eeca5048a496498ca1f (DEFAULT_TABLE table=default.Table_6175fc5b75b74fe5a7abe8861a2df365 [id=1d09f5b9fa164ac783f0be9332eac4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.616683 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 3802bec60e074107b9a8f8ddc3895fe2 (DEFAULT_TABLE table=default.Table_1b8997fc8d6b4f788983c76e84b2d0c5 [id=37382ab63e7d448788deaa4cd95c8b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.616824 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 652112d5039943628550188522bf0a17 with delete_type TABLET_DATA_DELETED (replaced by tablet 701f13f90f154e378b353e84d1bebd4e) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.617040 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82315e4c5a554eeca5048a496498ca1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.617093 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3802bec60e074107b9a8f8ddc3895fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.617152 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7c60ae6cebc04434975a2c3782f4aaa7" } modify_external_catalogs: true
I20251026 02:56:20.617266 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 20c39a62bfab429bb231ae073585d0e5 (DEFAULT_TABLE table=default.Table_0aa77747f2be46d8be9e7c5873940ca1 [id=a414f870eb244e0ba5d2569c6c91edc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.617309 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7c60ae6cebc04434975a2c3782f4aaa7" } modify_external_catalogs: true
I20251026 02:56:20.617611 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c39a62bfab429bb231ae073585d0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.618774 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1b8997fc8d6b4f788983c76e84b2d0c5" } modify_external_catalogs: true
I20251026 02:56:20.618944 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1b8997fc8d6b4f788983c76e84b2d0c5" } modify_external_catalogs: true
I20251026 02:56:20.620434 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_233cc7c9cac543889a30f7a5a8d22156" } modify_external_catalogs: true
I20251026 02:56:20.620532 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 3802bec60e074107b9a8f8ddc3895fe2 (DEFAULT_TABLE table=default.Table_1b8997fc8d6b4f788983c76e84b2d0c5 [id=37382ab63e7d448788deaa4cd95c8b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.620590 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_233cc7c9cac543889a30f7a5a8d22156" } modify_external_catalogs: true
I20251026 02:56:20.620886 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3802bec60e074107b9a8f8ddc3895fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.621415 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 760e6c761a7b49f695308fa1da9eda37 (DEFAULT_TABLE table=default.Table_664f6e878825452ebaeaed2a7a0aefc0 [id=c3e9c585b810403f88e4eb2b4924b73c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.621754 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760e6c761a7b49f695308fa1da9eda37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.624367 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 652112d5039943628550188522bf0a17 with delete_type TABLET_DATA_DELETED (replaced by tablet 701f13f90f154e378b353e84d1bebd4e) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.624840 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 20c39a62bfab429bb231ae073585d0e5 (DEFAULT_TABLE table=default.Table_0aa77747f2be46d8be9e7c5873940ca1 [id=a414f870eb244e0ba5d2569c6c91edc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.625171 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c39a62bfab429bb231ae073585d0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.627730 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 20c39a62bfab429bb231ae073585d0e5
I20251026 02:56:20.628579 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 82315e4c5a554eeca5048a496498ca1f (DEFAULT_TABLE table=default.Table_6175fc5b75b74fe5a7abe8861a2df365 [id=1d09f5b9fa164ac783f0be9332eac4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.628963 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82315e4c5a554eeca5048a496498ca1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.634800 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 82315e4c5a554eeca5048a496498ca1f (DEFAULT_TABLE table=default.Table_6175fc5b75b74fe5a7abe8861a2df365 [id=1d09f5b9fa164ac783f0be9332eac4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.635279 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82315e4c5a554eeca5048a496498ca1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.636073 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (DEFAULT_TABLE table=default.Table_cfef4f9714bd4b77b3a9c61ccbc25684 [id=095ad208529348d4aea2d4e2840e07c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.636370 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 20c39a62bfab429bb231ae073585d0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:20.636440 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bc92f90a68b4e749bfd4fa49076bfe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.636421 2092 tablet_service.cc:1505] Processing CreateTablet for tablet cd8644f86972403b8477f97e69be6f80 (DEFAULT_TABLE table=default.Table_d15145ea85264fc9aab94645c7ff3b41 [id=9d7e6bf7d8844942a0221a92da270f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.636840 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd8644f86972403b8477f97e69be6f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.637159 2095 tablet_service.cc:1505] Processing CreateTablet for tablet b7154ef047d94a8ebc305bc3b08d108c (DEFAULT_TABLE table=default.Table_c6bad2c857bb4c9fb8f8e2f5a87344cb [id=6fd49e190e4b4b2cba4b2fc5744b9f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.637494 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7154ef047d94a8ebc305bc3b08d108c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.638602 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_448f8c6a1d0c4125af909cf6570e41ae" } modify_external_catalogs: true
I20251026 02:56:20.638818 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_448f8c6a1d0c4125af909cf6570e41ae" } modify_external_catalogs: true
I20251026 02:56:20.639119 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 760e6c761a7b49f695308fa1da9eda37 (DEFAULT_TABLE table=default.Table_664f6e878825452ebaeaed2a7a0aefc0 [id=c3e9c585b810403f88e4eb2b4924b73c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.639454 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760e6c761a7b49f695308fa1da9eda37. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:20.640548 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 20c39a62bfab429bb231ae073585d0e5 because the tablet was not found. No further retry: Not found: Tablet not found: 20c39a62bfab429bb231ae073585d0e5
I20251026 02:56:20.642325 1624 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu pid 1771
I20251026 02:56:20.642877 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 20c39a62bfab429bb231ae073585d0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.644086 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 760e6c761a7b49f695308fa1da9eda37 (DEFAULT_TABLE table=default.Table_664f6e878825452ebaeaed2a7a0aefc0 [id=c3e9c585b810403f88e4eb2b4924b73c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.644493 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760e6c761a7b49f695308fa1da9eda37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.645943 2243 tablet_service.cc:1505] Processing CreateTablet for tablet b7154ef047d94a8ebc305bc3b08d108c (DEFAULT_TABLE table=default.Table_c6bad2c857bb4c9fb8f8e2f5a87344cb [id=6fd49e190e4b4b2cba4b2fc5744b9f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.646360 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7154ef047d94a8ebc305bc3b08d108c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.646929 1959 tablet_service.cc:1505] Processing CreateTablet for tablet cd8644f86972403b8477f97e69be6f80 (DEFAULT_TABLE table=default.Table_d15145ea85264fc9aab94645c7ff3b41 [id=9d7e6bf7d8844942a0221a92da270f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.647325 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd8644f86972403b8477f97e69be6f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.648594 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (DEFAULT_TABLE table=default.Table_cfef4f9714bd4b77b3a9c61ccbc25684 [id=095ad208529348d4aea2d4e2840e07c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.648998 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bc92f90a68b4e749bfd4fa49076bfe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.650151 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3802bec60e074107b9a8f8ddc3895fe2
I20251026 02:56:20.650897 1660 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 701f13f90f154e378b353e84d1bebd4e
I20251026 02:56:20.651290 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3f4436f64997445aa880610c5a93f955
I20251026 02:56:20.652776 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:20.653085 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (DEFAULT_TABLE table=default.Table_cfef4f9714bd4b77b3a9c61ccbc25684 [id=095ad208529348d4aea2d4e2840e07c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.653479 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bc92f90a68b4e749bfd4fa49076bfe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.653898 1835 generic_service.cc:196] Checking for leaks (request via RPC)
I20251026 02:56:20.656347 2309 heartbeater.cc:499] Master 127.1.150.62:44525 was elected leader, sending a full tablet report...
I20251026 02:56:20.656700 2308 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:20.658788 1656 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 630beab92a0d4b928c79d9f0c32795f0
I20251026 02:56:20.660408 2165 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
W20251026 02:56:20.663841 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 20c39a62bfab429bb231ae073585d0e5 because the tablet was not found. No further retry: Not found: Tablet not found: 20c39a62bfab429bb231ae073585d0e5
I20251026 02:56:20.664552 2381 consensus_queue.cc:260] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.666114 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f4436f64997445aa880610c5a93f955 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:20.668665 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3f4436f64997445aa880610c5a93f955 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f4436f64997445aa880610c5a93f955 already in progress: creating tablet
I20251026 02:56:20.669139 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 3fd66abba7364951a7659011c71e76af (DEFAULT_TABLE table=default.Table_99b42c4a4a874b68bb4beba4c08cdeb0 [id=472d44e7e6f0409ab9d238c4913cc8a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.669248 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 3802bec60e074107b9a8f8ddc3895fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.669586 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd66abba7364951a7659011c71e76af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.671507 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 64d00ae95e14439eaa748d17ca67118a (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.672050 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d00ae95e14439eaa748d17ca67118a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.672363 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 3802bec60e074107b9a8f8ddc3895fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.674065 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 3fd66abba7364951a7659011c71e76af (DEFAULT_TABLE table=default.Table_99b42c4a4a874b68bb4beba4c08cdeb0 [id=472d44e7e6f0409ab9d238c4913cc8a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.674427 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd66abba7364951a7659011c71e76af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.674518 1656 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 604b7272209d4652989c3e6d8201a255 (table default.Table_e4d2256ca9d340b78fe5fb50644c789a [id=d7ac91b3b98441baaa6ecc48cb904c7e]). Expected version 1 got 0
I20251026 02:56:20.674810 1656 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet de24439a6a3a4734a08160cb7393429a (table default.Table_3c222bf9316b4ffe89d6899f4ded8349 [id=931430a4a03741bd86175eabf9764a77]). Expected version 1 got 0
I20251026 02:56:20.674940 1656 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 0d066a461d9e4df198968c2e6f11ae2d (table default.Table_a445cc1241b548788d59b8dc29fa2f2b [id=3d118d55d42047bdbc0b8fb944b3a190]). Expected version 2 got 0
I20251026 02:56:20.676153 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 64d00ae95e14439eaa748d17ca67118a (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:20.676368 1647 catalog_manager.cc:4952] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3802bec60e074107b9a8f8ddc3895fe2 because the tablet was not found. No further retry: Not found: Tablet not found: 3802bec60e074107b9a8f8ddc3895fe2
I20251026 02:56:20.676507 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d00ae95e14439eaa748d17ca67118a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:20.676755 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3802bec60e074107b9a8f8ddc3895fe2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3802bec60e074107b9a8f8ddc3895fe2 already in progress: creating tablet
I20251026 02:56:20.669973 2234 tablet_service.cc:1505] Processing CreateTablet for tablet cd8644f86972403b8477f97e69be6f80 (DEFAULT_TABLE table=default.Table_d15145ea85264fc9aab94645c7ff3b41 [id=9d7e6bf7d8844942a0221a92da270f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.679297 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd8644f86972403b8477f97e69be6f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.680477 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f4436f64997445aa880610c5a93f955 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.680557 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b68c0b860984df99a7eeb73854a5da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.681056 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 20c39a62bfab429bb231ae073585d0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:20.681313 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0b68c0b860984df99a7eeb73854a5da8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b68c0b860984df99a7eeb73854a5da8 already in progress: creating tablet
I20251026 02:56:20.683247 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet ac43df5558634550a12aa65c521d8a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:20.684417 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ac43df5558634550a12aa65c521d8a14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac43df5558634550a12aa65c521d8a14 already in progress: creating tablet
W20251026 02:56:20.684619 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3f4436f64997445aa880610c5a93f955 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f4436f64997445aa880610c5a93f955 already in progress: creating tablet
W20251026 02:56:20.684863 1647 catalog_manager.cc:4952] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 20c39a62bfab429bb231ae073585d0e5 because the tablet was not found. No further retry: Not found: Tablet not found: 20c39a62bfab429bb231ae073585d0e5
I20251026 02:56:20.689355 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet deb5e70eb5f14e2ab260691ce79176de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.689639 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 652112d5039943628550188522bf0a17 with delete_type TABLET_DATA_DELETED (replaced by tablet 701f13f90f154e378b353e84d1bebd4e) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.689858 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f4436f64997445aa880610c5a93f955 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:20.690831 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet deb5e70eb5f14e2ab260691ce79176de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deb5e70eb5f14e2ab260691ce79176de already in progress: creating tablet
W20251026 02:56:20.691105 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 652112d5039943628550188522bf0a17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 652112d5039943628550188522bf0a17 already in progress: creating tablet
W20251026 02:56:20.691347 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3f4436f64997445aa880610c5a93f955 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f4436f64997445aa880610c5a93f955 already in progress: creating tablet
I20251026 02:56:20.691720 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4ba66147c249728713a42d20f541f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 630beab92a0d4b928c79d9f0c32795f0) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.692248 1962 tablet_service.cc:1505] Processing CreateTablet for tablet b7154ef047d94a8ebc305bc3b08d108c (DEFAULT_TABLE table=default.Table_c6bad2c857bb4c9fb8f8e2f5a87344cb [id=6fd49e190e4b4b2cba4b2fc5744b9f63]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:20.692339 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3a4ba66147c249728713a42d20f541f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a4ba66147c249728713a42d20f541f4 already in progress: creating tablet
I20251026 02:56:20.692582 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 3802bec60e074107b9a8f8ddc3895fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.692893 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7154ef047d94a8ebc305bc3b08d108c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:20.693184 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3802bec60e074107b9a8f8ddc3895fe2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3802bec60e074107b9a8f8ddc3895fe2 already in progress: creating tablet
I20251026 02:56:20.701125 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 652112d5039943628550188522bf0a17 with delete_type TABLET_DATA_DELETED (replaced by tablet 701f13f90f154e378b353e84d1bebd4e) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:20.701786 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 652112d5039943628550188522bf0a17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 652112d5039943628550188522bf0a17 already in progress: creating tablet
I20251026 02:56:20.703567 2381 ts_tablet_manager.cc:1434] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.108s user 0.015s sys 0.000s
I20251026 02:56:20.609365 2378 consensus_queue.cc:260] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.705663 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f4436f64997445aa880610c5a93f955 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:20.705971 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 3802bec60e074107b9a8f8ddc3895fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:20.706882 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3f4436f64997445aa880610c5a93f955 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f4436f64997445aa880610c5a93f955 already in progress: creating tablet
W20251026 02:56:20.707193 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3802bec60e074107b9a8f8ddc3895fe2 because the tablet was not found. No further retry: Not found: Tablet not found: 3802bec60e074107b9a8f8ddc3895fe2
I20251026 02:56:20.707738 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 3fd66abba7364951a7659011c71e76af (DEFAULT_TABLE table=default.Table_99b42c4a4a874b68bb4beba4c08cdeb0 [id=472d44e7e6f0409ab9d238c4913cc8a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.711398 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 652112d5039943628550188522bf0a17 with delete_type TABLET_DATA_DELETED (replaced by tablet 701f13f90f154e378b353e84d1bebd4e) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.711774 2377 tablet_replica.cc:333] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:20.712450 2377 raft_consensus.cc:2243] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:20.712918 2377 raft_consensus.cc:2272] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:20.714407 2377 ts_tablet_manager.cc:1916] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:20.717191 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f26f32ab73e848fea8718bfddbebb31f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:20.717598 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3ee177e686e54f8cabf3ccc9c8e6633f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.717897 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f26f32ab73e848fea8718bfddbebb31f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:20.718080 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ee177e686e54f8cabf3ccc9c8e6633f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.719264 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_71bb5e64488b4f4f92c57f6a1e34fb3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.719758 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71bb5e64488b4f4f92c57f6a1e34fb3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.721120 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_45490da4be10478b87d191904ec285e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.721695 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45490da4be10478b87d191904ec285e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.722775 2377 ts_tablet_manager.cc:1929] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:20.722915 2377 log.cc:1199] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/652112d5039943628550188522bf0a17
I20251026 02:56:20.723356 2377 ts_tablet_manager.cc:1950] T 652112d5039943628550188522bf0a17 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:20.724007 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 652112d5039943628550188522bf0a17 (table default.Table_233cc7c9cac543889a30f7a5a8d22156 [id=01000ef9384644bdb76c6cba086b605d]) successfully deleted
I20251026 02:56:20.724155 2378 ts_tablet_manager.cc:1434] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.126s user 0.011s sys 0.004s
I20251026 02:56:20.725847 2381 tablet_bootstrap.cc:492] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:20.727460 2381 tablet_bootstrap.cc:654] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:20.719318 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d89d54d7dc874d7b8d7599cbe59b281a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.728732 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d89d54d7dc874d7b8d7599cbe59b281a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.730733 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4ba66147c249728713a42d20f541f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 630beab92a0d4b928c79d9f0c32795f0) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.731062 2376 tablet_replica.cc:333] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:20.731551 2376 raft_consensus.cc:2243] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:20.731983 2376 raft_consensus.cc:2272] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:20.732795 2378 tablet_bootstrap.cc:492] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:20.734560 2378 tablet_bootstrap.cc:654] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:20.737758 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_995f3f5e6b574a93a90aa16e9543e458"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.738303 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_995f3f5e6b574a93a90aa16e9543e458 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.739372 2376 ts_tablet_manager.cc:1916] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:20.741390 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_382e6508ce4645699117b0e65595251c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:20.741950 2378 tablet_bootstrap.cc:492] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
W20251026 02:56:20.741992 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_382e6508ce4645699117b0e65595251c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.742093 2378 ts_tablet_manager.cc:1403] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20251026 02:56:20.743104 2378 raft_consensus.cc:359] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.743307 2378 raft_consensus.cc:385] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:20.743404 2378 raft_consensus.cc:740] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:20.743634 2378 consensus_queue.cc:260] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.744040 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2d81f1d4f7414e4b876393414dcedea7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.744832 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d81f1d4f7414e4b876393414dcedea7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.746446 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e8fa30fb54bb4950ae20643bf37e4aeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.746968 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8fa30fb54bb4950ae20643bf37e4aeb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.748344 2378 ts_tablet_manager.cc:1434] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251026 02:56:20.748483 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_81ca5851f387446ead27f921baefbe9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:20.748623 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f4436f64997445aa880610c5a93f955 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:20.748996 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81ca5851f387446ead27f921baefbe9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.752432 2378 tablet_bootstrap.cc:492] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:20.753345 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 3a4ba66147c249728713a42d20f541f4 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:20.753764 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet cd8644f86972403b8477f97e69be6f80 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:20.753729 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 64d00ae95e14439eaa748d17ca67118a (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.754806 2378 tablet_bootstrap.cc:654] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:20.764055 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd66abba7364951a7659011c71e76af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.768800 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d00ae95e14439eaa748d17ca67118a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.780931 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 3802bec60e074107b9a8f8ddc3895fe2 (DEFAULT_TABLE table=default.Table_1b8997fc8d6b4f788983c76e84b2d0c5 [id=37382ab63e7d448788deaa4cd95c8b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.781862 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3802bec60e074107b9a8f8ddc3895fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.783764 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e4d2256ca9d340b78fe5fb50644c789a" } new_table_name: "default.Table_e00202a12c9349eda8d47f735e4cddab" modify_external_catalogs: true
I20251026 02:56:20.785465 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_995f3f5e6b574a93a90aa16e9543e458" } new_table_name: "default.Table_1466d8b694c74b75bbc0b928e917993a" modify_external_catalogs: true
I20251026 02:56:20.786917 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_45490da4be10478b87d191904ec285e5" } new_table_name: "default.Table_ff61ac36382f428598c86ecb81001e88" modify_external_catalogs: true
I20251026 02:56:20.789919 1663 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 0d066a461d9e4df198968c2e6f11ae2d (table default.Table_a445cc1241b548788d59b8dc29fa2f2b [id=3d118d55d42047bdbc0b8fb944b3a190]). Expected version 2 got 0
I20251026 02:56:20.790560 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 19edc0546fd04110b37353178da2652e (DEFAULT_TABLE table=default.Table_f26f32ab73e848fea8718bfddbebb31f [id=aa6a3819787c4ab881cb8d71460a9b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.791019 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19edc0546fd04110b37353178da2652e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.792189 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 19edc0546fd04110b37353178da2652e (DEFAULT_TABLE table=default.Table_f26f32ab73e848fea8718bfddbebb31f [id=aa6a3819787c4ab881cb8d71460a9b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.792668 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19edc0546fd04110b37353178da2652e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.798466 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 19edc0546fd04110b37353178da2652e (DEFAULT_TABLE table=default.Table_f26f32ab73e848fea8718bfddbebb31f [id=aa6a3819787c4ab881cb8d71460a9b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.798914 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19edc0546fd04110b37353178da2652e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.802518 2376 ts_tablet_manager.cc:1929] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:20.803396 2376 log.cc:1199] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3a4ba66147c249728713a42d20f541f4
I20251026 02:56:20.803864 2381 tablet_bootstrap.cc:492] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:20.804039 2381 ts_tablet_manager.cc:1403] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.018s
I20251026 02:56:20.804479 2376 ts_tablet_manager.cc:1950] T 3a4ba66147c249728713a42d20f541f4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:20.804991 2381 raft_consensus.cc:359] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.805184 2381 raft_consensus.cc:385] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:20.805285 2381 raft_consensus.cc:740] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:20.805591 2381 consensus_queue.cc:260] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.806118 2381 ts_tablet_manager.cc:1434] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:20.806403 2381 tablet_bootstrap.cc:492] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:20.807706 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3a4ba66147c249728713a42d20f541f4 (table default.Table_448f8c6a1d0c4125af909cf6570e41ae [id=9e895e2eb27749839df30c205507d222]) successfully deleted
W20251026 02:56:20.807760 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 604b7272209d4652989c3e6d8201a255 consensus state has no leader
I20251026 02:56:20.808069 2381 tablet_bootstrap.cc:654] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:20.816649 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3ee177e686e54f8cabf3ccc9c8e6633f" } new_table_name: "default.Table_a5ff6167974b47798ffe2b3cd583b50b" modify_external_catalogs: true
I20251026 02:56:20.817483 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cfef4f9714bd4b77b3a9c61ccbc25684" } new_table_name: "default.Table_537013d780884ee89ee2ad679f506a49" modify_external_catalogs: true
I20251026 02:56:20.818771 2376 tablet_replica.cc:333] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:20.818987 2376 raft_consensus.cc:2243] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:20.819176 2376 raft_consensus.cc:2272] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:20.819990 2376 ts_tablet_manager.cc:1916] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:20.820461 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 64d00ae95e14439eaa748d17ca67118a from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:20.821306 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7fe6cbe999c54b299d729e97f0f938e7 has no consensus state
I20251026 02:56:20.822726 2378 tablet_bootstrap.cc:492] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:20.822908 2378 ts_tablet_manager.cc:1403] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.028s
I20251026 02:56:20.823678 2378 raft_consensus.cc:359] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.823871 2378 raft_consensus.cc:385] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:20.823949 2378 raft_consensus.cc:740] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:20.824262 2378 consensus_queue.cc:260] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.825811 2378 ts_tablet_manager.cc:1434] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:20.826138 2378 tablet_bootstrap.cc:492] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:20.827728 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet deb5e70eb5f14e2ab260691ce79176de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.828063 2378 tablet_bootstrap.cc:654] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:20.829557 2376 ts_tablet_manager.cc:1929] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:20.829700 2376 log.cc:1199] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3f4436f64997445aa880610c5a93f955
I20251026 02:56:20.830099 2376 ts_tablet_manager.cc:1950] T 3f4436f64997445aa880610c5a93f955 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:20.831131 2376 tablet_replica.cc:333] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:20.831068 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3f4436f64997445aa880610c5a93f955 (table default.Table_7c60ae6cebc04434975a2c3782f4aaa7 [id=9d77f9be68084772a1f3ee4d00a5180d]) successfully deleted
I20251026 02:56:20.831303 2376 raft_consensus.cc:2243] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:20.831509 2376 raft_consensus.cc:2272] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:20.832417 2376 ts_tablet_manager.cc:1916] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:20.833194 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 426ce6fb80874df68210a652f7b7602f has no consensus state
I20251026 02:56:20.837852 2375 ts_tablet_manager.cc:1434] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.258s user 0.024s sys 0.050s
I20251026 02:56:20.840548 2375 tablet_bootstrap.cc:492] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:20.841318 2378 tablet_bootstrap.cc:492] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:20.841496 2378 ts_tablet_manager.cc:1403] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.003s
I20251026 02:56:20.842355 2378 raft_consensus.cc:359] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.842553 2378 raft_consensus.cc:385] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:20.842653 2375 tablet_bootstrap.cc:654] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:20.842649 2378 raft_consensus.cc:740] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:20.842890 2378 consensus_queue.cc:260] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.843415 2378 ts_tablet_manager.cc:1434] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:20.843731 2378 tablet_bootstrap.cc:492] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:20.845451 2378 tablet_bootstrap.cc:654] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:20.846414 2376 ts_tablet_manager.cc:1929] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:20.846522 2376 log.cc:1199] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/deb5e70eb5f14e2ab260691ce79176de
I20251026 02:56:20.846876 2376 ts_tablet_manager.cc:1950] T deb5e70eb5f14e2ab260691ce79176de P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:20.847224 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c6bad2c857bb4c9fb8f8e2f5a87344cb" } modify_external_catalogs: true
I20251026 02:56:20.847406 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c6bad2c857bb4c9fb8f8e2f5a87344cb" } modify_external_catalogs: true
I20251026 02:56:20.849182 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet deb5e70eb5f14e2ab260691ce79176de (table default.Table_f070d68ce8e342ffb918288c7b46f3f8 [id=c1d3e3f0c2b7403387d77c9efcde8e7d]) successfully deleted
I20251026 02:56:20.850085 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b68c0b860984df99a7eeb73854a5da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.853567 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet b7154ef047d94a8ebc305bc3b08d108c from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:20.857921 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79942752cba4429abdc9524f882800da has no consensus state
I20251026 02:56:20.858842 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet 19edc0546fd04110b37353178da2652e from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:20.858893 2376 tablet_replica.cc:333] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:20.859187 2376 raft_consensus.cc:2243] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:20.859412 2376 raft_consensus.cc:2272] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:20.859668 1660 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 0d066a461d9e4df198968c2e6f11ae2d (table default.Table_a445cc1241b548788d59b8dc29fa2f2b [id=3d118d55d42047bdbc0b8fb944b3a190]). Expected version 2 got 0
I20251026 02:56:20.860766 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 0d066a461d9e4df198968c2e6f11ae2d (table default.Table_a445cc1241b548788d59b8dc29fa2f2b [id=3d118d55d42047bdbc0b8fb944b3a190]). Expected version 2 got 0
I20251026 02:56:20.861722 2376 ts_tablet_manager.cc:1916] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:20.866541 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 9a04ac20b7624a018c403ef4f64f7a69 (DEFAULT_TABLE table=default.Table_2d81f1d4f7414e4b876393414dcedea7 [id=c162ef89d2f14e91a24a6d8c2c37c261]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.867115 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a04ac20b7624a018c403ef4f64f7a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.874069 2392 raft_consensus.cc:493] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:20.874464 2168 tablet.cc:2378] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 02:56:20.874399 2392 raft_consensus.cc:515] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.883121 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 9a04ac20b7624a018c403ef4f64f7a69 (DEFAULT_TABLE table=default.Table_2d81f1d4f7414e4b876393414dcedea7 [id=c162ef89d2f14e91a24a6d8c2c37c261]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:20.883517 2311 tablet.cc:2378] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 02:56:20.883733 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a04ac20b7624a018c403ef4f64f7a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.887914 2392 leader_election.cc:290] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:20.897132 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 79942752cba4429abdc9524f882800da (DEFAULT_TABLE table=default.Table_a5ff6167974b47798ffe2b3cd583b50b [id=43cb7f538f824ae38059645230cc7e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.897750 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79942752cba4429abdc9524f882800da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.897943 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 79942752cba4429abdc9524f882800da (DEFAULT_TABLE table=default.Table_a5ff6167974b47798ffe2b3cd583b50b [id=43cb7f538f824ae38059645230cc7e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.898303 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79942752cba4429abdc9524f882800da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.900550 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 5e56e5eec1474aefae04d278ded30765 (DEFAULT_TABLE table=default.Table_e8fa30fb54bb4950ae20643bf37e4aeb [id=6734e5731cc94e71b43c76512f649d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.900923 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e56e5eec1474aefae04d278ded30765. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.902302 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 4eea9b18e5094a22a51942f438a20eb4 (DEFAULT_TABLE table=default.Table_71bb5e64488b4f4f92c57f6a1e34fb3c [id=30a69f019af84e09af9f21939f081847]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.902663 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eea9b18e5094a22a51942f438a20eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.904592 1960 tablet_service.cc:1505] Processing CreateTablet for tablet e79223b1086347199470a2526e32bd6f (DEFAULT_TABLE table=default.Table_382e6508ce4645699117b0e65595251c [id=214ae71fbaf24f4ab016cb643a89f97f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.904984 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e79223b1086347199470a2526e32bd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.905395 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 5e56e5eec1474aefae04d278ded30765 (DEFAULT_TABLE table=default.Table_e8fa30fb54bb4950ae20643bf37e4aeb [id=6734e5731cc94e71b43c76512f649d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.905790 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e56e5eec1474aefae04d278ded30765. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.908505 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 4eea9b18e5094a22a51942f438a20eb4 (DEFAULT_TABLE table=default.Table_71bb5e64488b4f4f92c57f6a1e34fb3c [id=30a69f019af84e09af9f21939f081847]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.908531 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 7c32c2393a1144e9950b00f61ddba4eb (DEFAULT_TABLE table=default.Table_d89d54d7dc874d7b8d7599cbe59b281a [id=85ee8f90981b4b4f899a45b6c29ff1e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.908869 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c32c2393a1144e9950b00f61ddba4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.908890 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eea9b18e5094a22a51942f438a20eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.912531 2096 tablet_service.cc:1505] Processing CreateTablet for tablet e79223b1086347199470a2526e32bd6f (DEFAULT_TABLE table=default.Table_382e6508ce4645699117b0e65595251c [id=214ae71fbaf24f4ab016cb643a89f97f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.912938 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e79223b1086347199470a2526e32bd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.913612 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 0d066a461d9e4df198968c2e6f11ae2d (table default.Table_a445cc1241b548788d59b8dc29fa2f2b [id=3d118d55d42047bdbc0b8fb944b3a190]). Expected version 2 got 0
W20251026 02:56:20.914382 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7bc92f90a68b4e749bfd4fa49076bfe2 consensus state has no leader
I20251026 02:56:20.919207 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d15145ea85264fc9aab94645c7ff3b41" } modify_external_catalogs: true
I20251026 02:56:20.919476 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d15145ea85264fc9aab94645c7ff3b41" } modify_external_catalogs: true
I20251026 02:56:20.920135 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e00202a12c9349eda8d47f735e4cddab" } modify_external_catalogs: true
I20251026 02:56:20.920398 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e00202a12c9349eda8d47f735e4cddab" } modify_external_catalogs: true
I20251026 02:56:20.922192 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ff61ac36382f428598c86ecb81001e88" } modify_external_catalogs: true
I20251026 02:56:20.922391 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ff61ac36382f428598c86ecb81001e88" } modify_external_catalogs: true
I20251026 02:56:20.924274 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_382e6508ce4645699117b0e65595251c" } modify_external_catalogs: true
I20251026 02:56:20.924468 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_382e6508ce4645699117b0e65595251c" } modify_external_catalogs: true
I20251026 02:56:20.932719 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 64d00ae95e14439eaa748d17ca67118a
I20251026 02:56:20.933389 1654 catalog_manager.cc:6719] ReplaceTablet: tablet 64d00ae95e14439eaa748d17ca67118a deleted and replaced by tablet 3ef7247d62ec4adc82fdbed40fc852ea
I20251026 02:56:20.933781 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet 64d00ae95e14439eaa748d17ca67118a with delete_type TABLET_DATA_DELETED (replaced by tablet 3ef7247d62ec4adc82fdbed40fc852ea) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.934963 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 64d00ae95e14439eaa748d17ca67118a with delete_type TABLET_DATA_DELETED (replaced by tablet 3ef7247d62ec4adc82fdbed40fc852ea) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:20.937436 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 64d00ae95e14439eaa748d17ca67118a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d00ae95e14439eaa748d17ca67118a already in progress: creating tablet
I20251026 02:56:20.941080 2381 tablet_bootstrap.cc:492] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:20.941649 2381 ts_tablet_manager.cc:1403] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.135s user 0.042s sys 0.033s
I20251026 02:56:20.943265 2381 raft_consensus.cc:359] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.945057 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1b3d02c229434ae4a351fd493a20654b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.945912 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b3d02c229434ae4a351fd493a20654b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.946708 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 9a04ac20b7624a018c403ef4f64f7a69 (DEFAULT_TABLE table=default.Table_2d81f1d4f7414e4b876393414dcedea7 [id=c162ef89d2f14e91a24a6d8c2c37c261]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.950403 2381 raft_consensus.cc:385] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:20.950496 2381 raft_consensus.cc:740] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:20.950701 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 64d00ae95e14439eaa748d17ca67118a with delete_type TABLET_DATA_DELETED (replaced by tablet 3ef7247d62ec4adc82fdbed40fc852ea) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.946985 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 79942752cba4429abdc9524f882800da (DEFAULT_TABLE table=default.Table_a5ff6167974b47798ffe2b3cd583b50b [id=43cb7f538f824ae38059645230cc7e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.947542 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 5e56e5eec1474aefae04d278ded30765 (DEFAULT_TABLE table=default.Table_e8fa30fb54bb4950ae20643bf37e4aeb [id=6734e5731cc94e71b43c76512f649d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.950728 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a04ac20b7624a018c403ef4f64f7a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.951424 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e56e5eec1474aefae04d278ded30765. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.947672 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 4eea9b18e5094a22a51942f438a20eb4 (DEFAULT_TABLE table=default.Table_71bb5e64488b4f4f92c57f6a1e34fb3c [id=30a69f019af84e09af9f21939f081847]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.955281 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eea9b18e5094a22a51942f438a20eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.955659 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet cd8644f86972403b8477f97e69be6f80
I20251026 02:56:20.956130 1658 catalog_manager.cc:6719] ReplaceTablet: tablet cd8644f86972403b8477f97e69be6f80 deleted and replaced by tablet 5ff86fc06e7d46e78fe68cc6f31a30cc
I20251026 02:56:20.958120 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet cd8644f86972403b8477f97e69be6f80 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ff86fc06e7d46e78fe68cc6f31a30cc) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:20.958819 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet cd8644f86972403b8477f97e69be6f80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd8644f86972403b8477f97e69be6f80 already in progress: creating tablet
I20251026 02:56:20.947996 2237 tablet_service.cc:1505] Processing CreateTablet for tablet e79223b1086347199470a2526e32bd6f (DEFAULT_TABLE table=default.Table_382e6508ce4645699117b0e65595251c [id=214ae71fbaf24f4ab016cb643a89f97f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.959434 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e79223b1086347199470a2526e32bd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.960587 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 7c32c2393a1144e9950b00f61ddba4eb (DEFAULT_TABLE table=default.Table_d89d54d7dc874d7b8d7599cbe59b281a [id=85ee8f90981b4b4f899a45b6c29ff1e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.960973 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c32c2393a1144e9950b00f61ddba4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.961578 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet cd8644f86972403b8477f97e69be6f80 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ff86fc06e7d46e78fe68cc6f31a30cc) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.962247 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_bcf0aac1b41747e9a94a97b8430fa081"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.962833 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcf0aac1b41747e9a94a97b8430fa081 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.963554 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet cd8644f86972403b8477f97e69be6f80 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ff86fc06e7d46e78fe68cc6f31a30cc) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:20.964052 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 64d00ae95e14439eaa748d17ca67118a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d00ae95e14439eaa748d17ca67118a already in progress: creating tablet
W20251026 02:56:20.964461 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet cd8644f86972403b8477f97e69be6f80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd8644f86972403b8477f97e69be6f80 already in progress: creating tablet
I20251026 02:56:20.964824 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_03c78b21a07841b2b1a8962abe80a10e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.965353 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03c78b21a07841b2b1a8962abe80a10e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.968250 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79942752cba4429abdc9524f882800da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.971546 2376 ts_tablet_manager.cc:1929] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:20.971731 2376 log.cc:1199] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/0b68c0b860984df99a7eeb73854a5da8
I20251026 02:56:20.971786 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b7154ef047d94a8ebc305bc3b08d108c
I20251026 02:56:20.972453 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet 20c39a62bfab429bb231ae073585d0e5 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:20.972918 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet b7154ef047d94a8ebc305bc3b08d108c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:20.973534 2376 ts_tablet_manager.cc:1950] T 0b68c0b860984df99a7eeb73854a5da8 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:20.975139 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet b7154ef047d94a8ebc305bc3b08d108c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:20.975785 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 0b68c0b860984df99a7eeb73854a5da8 (table default.Table_0d997fa5825e4cf2a797881f912b609f [id=319c8d881ba442c6a0629a979e01b289]) successfully deleted
W20251026 02:56:20.976058 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 64d00ae95e14439eaa748d17ca67118a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d00ae95e14439eaa748d17ca67118a already in progress: creating tablet
W20251026 02:56:20.976305 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet b7154ef047d94a8ebc305bc3b08d108c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7154ef047d94a8ebc305bc3b08d108c already in progress: creating tablet
W20251026 02:56:20.977113 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet b7154ef047d94a8ebc305bc3b08d108c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7154ef047d94a8ebc305bc3b08d108c already in progress: creating tablet
I20251026 02:56:20.977919 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet b7154ef047d94a8ebc305bc3b08d108c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:20.978379 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ad061592630140d9952cf22072721a4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:20.979001 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad061592630140d9952cf22072721a4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.980262 2381 consensus_queue.cc:260] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.980775 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d066a461d9e4df198968c2e6f11ae2d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:20.981484 2375 tablet_bootstrap.cc:492] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:20.981638 2375 ts_tablet_manager.cc:1403] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.141s user 0.003s sys 0.000s
I20251026 02:56:20.982931 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 426ce6fb80874df68210a652f7b7602f (DEFAULT_TABLE table=default.Table_ff61ac36382f428598c86ecb81001e88 [id=6071000b0280484b8be3da135c7f7b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.983369 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426ce6fb80874df68210a652f7b7602f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.983911 2381 ts_tablet_manager.cc:1434] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.042s user 0.000s sys 0.002s
I20251026 02:56:20.984342 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d066a461d9e4df198968c2e6f11ae2d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:20.984920 2120 raft_consensus.cc:2468] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:20.985406 2381 tablet_bootstrap.cc:492] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:20.987142 2381 tablet_bootstrap.cc:654] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:20.987200 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet cd8644f86972403b8477f97e69be6f80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd8644f86972403b8477f97e69be6f80 already in progress: creating tablet
W20251026 02:56:20.987432 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b7154ef047d94a8ebc305bc3b08d108c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7154ef047d94a8ebc305bc3b08d108c already in progress: creating tablet
I20251026 02:56:20.988370 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_cf4652bce4e447caa066bfebc7b43e50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:20.988577 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 7fe6cbe999c54b299d729e97f0f938e7 (DEFAULT_TABLE table=default.Table_1466d8b694c74b75bbc0b928e917993a [id=53cd517e68e64597ad51d85d16f0399d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.988714 2195 leader_election.cc:304] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:20.988931 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fe6cbe999c54b299d729e97f0f938e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:20.988936 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf4652bce4e447caa066bfebc7b43e50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:20.989166 2392 raft_consensus.cc:2804] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:20.989341 2392 raft_consensus.cc:493] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:20.989368 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 426ce6fb80874df68210a652f7b7602f (DEFAULT_TABLE table=default.Table_ff61ac36382f428598c86ecb81001e88 [id=6071000b0280484b8be3da135c7f7b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.989451 2392 raft_consensus.cc:3060] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:20.989770 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426ce6fb80874df68210a652f7b7602f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:20.992760 2392 raft_consensus.cc:515] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.993543 2392 leader_election.cc:290] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:20.995090 2375 raft_consensus.cc:359] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.995651 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 426ce6fb80874df68210a652f7b7602f (DEFAULT_TABLE table=default.Table_ff61ac36382f428598c86ecb81001e88 [id=6071000b0280484b8be3da135c7f7b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.995887 2375 raft_consensus.cc:385] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:20.995249 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d066a461d9e4df198968c2e6f11ae2d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:20.996060 2375 raft_consensus.cc:740] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:20.996155 2120 raft_consensus.cc:3060] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:20.996354 2375 consensus_queue.cc:260] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:20.996500 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 7fe6cbe999c54b299d729e97f0f938e7 (DEFAULT_TABLE table=default.Table_1466d8b694c74b75bbc0b928e917993a [id=53cd517e68e64597ad51d85d16f0399d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:20.996887 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fe6cbe999c54b299d729e97f0f938e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:20.996933 2194 leader_election.cc:343] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:20.999352 2120 raft_consensus.cc:2468] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:21.000358 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 19edc0546fd04110b37353178da2652e
I20251026 02:56:20.999917 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d066a461d9e4df198968c2e6f11ae2d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:21.000902 1662 catalog_manager.cc:6719] ReplaceTablet: tablet 19edc0546fd04110b37353178da2652e deleted and replaced by tablet ddfe9452fbe4426fb8f955b4bed586d9
W20251026 02:56:21.001132 2194 leader_election.cc:343] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.001523 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 19edc0546fd04110b37353178da2652e with delete_type TABLET_DATA_DELETED (replaced by tablet ddfe9452fbe4426fb8f955b4bed586d9) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.002154 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 19edc0546fd04110b37353178da2652e with delete_type TABLET_DATA_DELETED (replaced by tablet ddfe9452fbe4426fb8f955b4bed586d9) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:21.002660 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 19edc0546fd04110b37353178da2652e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19edc0546fd04110b37353178da2652e already in progress: creating tablet
I20251026 02:56:21.002707 2195 leader_election.cc:304] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:21.003275 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 19edc0546fd04110b37353178da2652e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19edc0546fd04110b37353178da2652e already in progress: creating tablet
I20251026 02:56:21.003361 2375 ts_tablet_manager.cc:1434] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.022s user 0.000s sys 0.003s
I20251026 02:56:21.003587 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426ce6fb80874df68210a652f7b7602f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.003631 2375 tablet_bootstrap.cc:492] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:21.003710 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 7c32c2393a1144e9950b00f61ddba4eb (DEFAULT_TABLE table=default.Table_d89d54d7dc874d7b8d7599cbe59b281a [id=85ee8f90981b4b4f899a45b6c29ff1e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.004076 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c32c2393a1144e9950b00f61ddba4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.005026 2392 raft_consensus.cc:2804] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:21.005415 2375 tablet_bootstrap.cc:654] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.008764 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet deb5e70eb5f14e2ab260691ce79176de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.009074 2377 tablet_replica.cc:333] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:21.009348 2377 raft_consensus.cc:2243] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.009555 2377 raft_consensus.cc:2272] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.009598 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ff247a755b524118a9e9fcb6f8745092"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:21.009917 2378 tablet_bootstrap.cc:492] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.010093 2378 ts_tablet_manager.cc:1403] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.166s user 0.037s sys 0.064s
W20251026 02:56:21.010185 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff247a755b524118a9e9fcb6f8745092 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.010946 2378 raft_consensus.cc:359] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.011147 2378 raft_consensus.cc:385] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.011220 2378 raft_consensus.cc:740] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.011479 2378 consensus_queue.cc:260] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.013051 2377 ts_tablet_manager.cc:1916] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.014580 2378 ts_tablet_manager.cc:1434] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:21.016508 2378 tablet_bootstrap.cc:492] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.018249 2378 tablet_bootstrap.cc:654] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.018954 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 604b7272209d4652989c3e6d8201a255
I20251026 02:56:21.019701 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7272209d4652989c3e6d8201a255 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.020630 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7272209d4652989c3e6d8201a255 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:21.021407 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 604b7272209d4652989c3e6d8201a255 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 604b7272209d4652989c3e6d8201a255 already in progress: creating tablet
I20251026 02:56:21.021747 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7272209d4652989c3e6d8201a255 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:21.022410 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 604b7272209d4652989c3e6d8201a255 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 604b7272209d4652989c3e6d8201a255 already in progress: creating tablet
I20251026 02:56:21.022523 2377 ts_tablet_manager.cc:1929] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.022663 2377 log.cc:1199] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/deb5e70eb5f14e2ab260691ce79176de
I20251026 02:56:21.022804 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 19edc0546fd04110b37353178da2652e with delete_type TABLET_DATA_DELETED (replaced by tablet ddfe9452fbe4426fb8f955b4bed586d9) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.023049 2377 ts_tablet_manager.cc:1950] T deb5e70eb5f14e2ab260691ce79176de P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:21.023437 2381 tablet_bootstrap.cc:492] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.023581 2381 ts_tablet_manager.cc:1403] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.003s
I20251026 02:56:21.024170 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet deb5e70eb5f14e2ab260691ce79176de (table default.Table_f070d68ce8e342ffb918288c7b46f3f8 [id=c1d3e3f0c2b7403387d77c9efcde8e7d]) successfully deleted
I20251026 02:56:21.024377 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet deb5e70eb5f14e2ab260691ce79176de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:21.024607 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 604b7272209d4652989c3e6d8201a255 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 604b7272209d4652989c3e6d8201a255 already in progress: creating tablet
I20251026 02:56:21.024727 2372 tablet_replica.cc:333] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:21.025213 2372 raft_consensus.cc:2243] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.025300 2375 tablet_bootstrap.cc:492] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:21.025337 2381 raft_consensus.cc:359] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.025482 2375 ts_tablet_manager.cc:1403] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.003s
I20251026 02:56:21.025501 2381 raft_consensus.cc:385] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.025580 2381 raft_consensus.cc:740] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.025633 2372 raft_consensus.cc:2272] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.025835 2381 consensus_queue.cc:260] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.026491 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet deb5e70eb5f14e2ab260691ce79176de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.026782 2375 raft_consensus.cc:359] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.027012 2375 raft_consensus.cc:385] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.027097 2375 raft_consensus.cc:740] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.027442 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_17277a07df7c41a3a3b95c89e0459a5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:21.027529 2372 ts_tablet_manager.cc:1916] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.027563 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 652112d5039943628550188522bf0a17 with delete_type TABLET_DATA_DELETED (replaced by tablet 701f13f90f154e378b353e84d1bebd4e) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:21.027998 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17277a07df7c41a3a3b95c89e0459a5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.028690 2375 consensus_queue.cc:260] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.029204 2375 ts_tablet_manager.cc:1434] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251026 02:56:21.030419 2378 tablet_bootstrap.cc:492] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.030589 2378 ts_tablet_manager.cc:1403] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20251026 02:56:21.030731 2376 tablet_replica.cc:333] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:21.030949 2376 raft_consensus.cc:2243] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.031212 2376 raft_consensus.cc:2272] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.031518 2378 raft_consensus.cc:359] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:21.031728 2378 raft_consensus.cc:385] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.031826 2378 raft_consensus.cc:740] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.032096 2378 consensus_queue.cc:260] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:21.032374 2376 ts_tablet_manager.cc:1916] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.035269 2378 ts_tablet_manager.cc:1434] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251026 02:56:21.037111 2378 tablet_bootstrap.cc:492] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.039916 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 7fe6cbe999c54b299d729e97f0f938e7 (DEFAULT_TABLE table=default.Table_1466d8b694c74b75bbc0b928e917993a [id=53cd517e68e64597ad51d85d16f0399d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.040403 2378 tablet_bootstrap.cc:654] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.045102 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 426ce6fb80874df68210a652f7b7602f
I20251026 02:56:21.045114 2372 ts_tablet_manager.cc:1929] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.045310 2372 log.cc:1199] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/deb5e70eb5f14e2ab260691ce79176de
I20251026 02:56:21.045779 2372 ts_tablet_manager.cc:1950] T deb5e70eb5f14e2ab260691ce79176de P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:21.045948 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 426ce6fb80874df68210a652f7b7602f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.047250 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 426ce6fb80874df68210a652f7b7602f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.047502 2392 raft_consensus.cc:697] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
W20251026 02:56:21.047567 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 19edc0546fd04110b37353178da2652e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19edc0546fd04110b37353178da2652e already in progress: creating tablet
W20251026 02:56:21.047827 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet deb5e70eb5f14e2ab260691ce79176de because the tablet was not found. No further retry: Not found: Tablet not found: deb5e70eb5f14e2ab260691ce79176de
W20251026 02:56:21.048066 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 426ce6fb80874df68210a652f7b7602f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 426ce6fb80874df68210a652f7b7602f already in progress: creating tablet
I20251026 02:56:21.048022 2392 consensus_queue.cc:237] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.049810 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 426ce6fb80874df68210a652f7b7602f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.051425 2378 tablet_bootstrap.cc:492] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.051616 2378 ts_tablet_manager.cc:1403] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.006s
I20251026 02:56:21.051652 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_50922eae951b497ba3b1e180c87facfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.052325 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50922eae951b497ba3b1e180c87facfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.052515 2378 raft_consensus.cc:359] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.052706 2378 raft_consensus.cc:385] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.052770 2378 raft_consensus.cc:740] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.053052 2378 consensus_queue.cc:260] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.055557 2378 ts_tablet_manager.cc:1434] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:21.057997 2378 tablet_bootstrap.cc:492] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.059685 2381 ts_tablet_manager.cc:1434] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.036s user 0.003s sys 0.013s
I20251026 02:56:21.059900 2378 tablet_bootstrap.cc:654] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.059942 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet e79223b1086347199470a2526e32bd6f
I20251026 02:56:21.060021 2381 tablet_bootstrap.cc:492] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.060842 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet e79223b1086347199470a2526e32bd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.061290 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet e79223b1086347199470a2526e32bd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:21.061949 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet e79223b1086347199470a2526e32bd6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e79223b1086347199470a2526e32bd6f already in progress: creating tablet
I20251026 02:56:21.064419 2381 tablet_bootstrap.cc:654] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.065578 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet e79223b1086347199470a2526e32bd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.069275 2378 tablet_bootstrap.cc:492] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.069489 2378 ts_tablet_manager.cc:1403] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20251026 02:56:21.070297 2378 raft_consensus.cc:359] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.070477 2378 raft_consensus.cc:385] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.070573 2378 raft_consensus.cc:740] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.070830 2378 consensus_queue.cc:260] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.071990 2381 tablet_bootstrap.cc:492] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.072156 2381 ts_tablet_manager.cc:1403] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.002s
I20251026 02:56:21.072553 2378 ts_tablet_manager.cc:1434] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:21.072893 2378 tablet_bootstrap.cc:492] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.073414 2381 raft_consensus.cc:359] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.073601 2381 raft_consensus.cc:385] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.073660 2381 raft_consensus.cc:740] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.073909 2381 consensus_queue.cc:260] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.074630 2378 tablet_bootstrap.cc:654] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.076480 2381 ts_tablet_manager.cc:1434] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251026 02:56:21.077751 2381 tablet_bootstrap.cc:492] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.079351 2381 tablet_bootstrap.cc:654] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.086194 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fe6cbe999c54b299d729e97f0f938e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.088009 2381 tablet_bootstrap.cc:492] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.088169 2381 ts_tablet_manager.cc:1403] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20251026 02:56:21.088940 2381 raft_consensus.cc:359] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.089135 2381 raft_consensus.cc:385] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.089218 2381 raft_consensus.cc:740] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.089449 2381 consensus_queue.cc:260] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.090883 2381 ts_tablet_manager.cc:1434] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 02:56:21.091161 2381 tablet_bootstrap.cc:492] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.092875 2381 tablet_bootstrap.cc:654] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.104455 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet deb5e70eb5f14e2ab260691ce79176de (table default.Table_f070d68ce8e342ffb918288c7b46f3f8 [id=c1d3e3f0c2b7403387d77c9efcde8e7d]) successfully deleted
W20251026 02:56:21.104859 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 426ce6fb80874df68210a652f7b7602f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 426ce6fb80874df68210a652f7b7602f already in progress: creating tablet
W20251026 02:56:21.105123 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet e79223b1086347199470a2526e32bd6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e79223b1086347199470a2526e32bd6f already in progress: creating tablet
I20251026 02:56:21.105159 2375 tablet_bootstrap.cc:492] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:21.107425 2375 tablet_bootstrap.cc:654] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.108651 2381 tablet_bootstrap.cc:492] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.108824 2381 ts_tablet_manager.cc:1403] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.005s
I20251026 02:56:21.109542 2381 raft_consensus.cc:359] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.109746 2381 raft_consensus.cc:385] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.109825 2381 raft_consensus.cc:740] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.110064 2381 consensus_queue.cc:260] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.110193 2376 ts_tablet_manager.cc:1929] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.110375 2376 log.cc:1199] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/652112d5039943628550188522bf0a17
I20251026 02:56:21.110833 2376 ts_tablet_manager.cc:1950] T 652112d5039943628550188522bf0a17 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:21.112037 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 652112d5039943628550188522bf0a17 (table default.Table_233cc7c9cac543889a30f7a5a8d22156 [id=01000ef9384644bdb76c6cba086b605d]) successfully deleted
I20251026 02:56:21.112532 2381 ts_tablet_manager.cc:1434] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:21.115114 2381 tablet_bootstrap.cc:492] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.116297 1659 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 5ff86fc06e7d46e78fe68cc6f31a30cc
I20251026 02:56:21.116767 2381 tablet_bootstrap.cc:654] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.120731 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_aa762aa5113a46a98da2f4ded4e23172"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.121328 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa762aa5113a46a98da2f4ded4e23172 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:21.122752 2030 tablet.cc:2378] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 02:56:21.125190 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f4436f64997445aa880610c5a93f955 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.125742 2372 tablet_replica.cc:333] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:21.125924 2372 raft_consensus.cc:2243] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.126082 2372 raft_consensus.cc:2272] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.126829 2372 ts_tablet_manager.cc:1916] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.135360 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_75c8f310e69c4237a406e4f6bd3f6b15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.135928 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75c8f310e69c4237a406e4f6bd3f6b15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.143402 2372 ts_tablet_manager.cc:1929] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.143661 2372 log.cc:1199] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3f4436f64997445aa880610c5a93f955
I20251026 02:56:21.144176 2372 ts_tablet_manager.cc:1950] T 3f4436f64997445aa880610c5a93f955 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:21.145334 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3f4436f64997445aa880610c5a93f955 (table default.Table_7c60ae6cebc04434975a2c3782f4aaa7 [id=9d77f9be68084772a1f3ee4d00a5180d]) successfully deleted
I20251026 02:56:21.150394 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a445cc1241b548788d59b8dc29fa2f2b" } new_table_name: "default.Table_0a8522a9059f4189b8cb07ff7409bd3a" modify_external_catalogs: true
W20251026 02:56:21.153391 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 426ce6fb80874df68210a652f7b7602f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 426ce6fb80874df68210a652f7b7602f already in progress: creating tablet
W20251026 02:56:21.154246 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet e79223b1086347199470a2526e32bd6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e79223b1086347199470a2526e32bd6f already in progress: creating tablet
I20251026 02:56:21.155793 2393 raft_consensus.cc:493] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.156067 2393 raft_consensus.cc:515] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:21.158110 2393 leader_election.cc:290] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.198668 1624 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu with pid 1771
I20251026 02:56:21.217407 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e8fa30fb54bb4950ae20643bf37e4aeb" } new_table_name: "default.Table_83bdf2cfef4042748553ebe8ae34996a" modify_external_catalogs: true
I20251026 02:56:21.226935 2393 raft_consensus.cc:493] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.227177 2393 raft_consensus.cc:515] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.227953 2393 leader_election.cc:290] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:21.238512 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6175fc5b75b74fe5a7abe8861a2df365" } new_table_name: "default.Table_cfeac8daaf08492a9456d2e06cd62a58" modify_external_catalogs: true
I20251026 02:56:21.240926 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2d81f1d4f7414e4b876393414dcedea7" } new_table_name: "default.Table_0d809da9aa7f4c8cbaf054721e6aa692" modify_external_catalogs: true
I20251026 02:56:21.241780 2378 tablet_bootstrap.cc:492] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.242018 2378 ts_tablet_manager.cc:1403] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.169s user 0.021s sys 0.061s
I20251026 02:56:21.243129 2378 raft_consensus.cc:359] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.243439 2378 raft_consensus.cc:385] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.243532 2378 raft_consensus.cc:740] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.244325 2378 consensus_queue.cc:260] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.245690 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_664f6e878825452ebaeaed2a7a0aefc0" } new_table_name: "default.Table_1854fa4e2e14427aadd889784db2cc44" modify_external_catalogs: true
I20251026 02:56:21.247001 2378 ts_tablet_manager.cc:1434] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251026 02:56:21.247388 2378 tablet_bootstrap.cc:492] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.251418 2392 raft_consensus.cc:493] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.251619 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 0d066a461d9e4df198968c2e6f11ae2d from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:21.251780 2375 tablet_bootstrap.cc:492] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:21.251675 2392 raft_consensus.cc:515] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.251936 2375 ts_tablet_manager.cc:1403] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.147s user 0.005s sys 0.000s
I20251026 02:56:21.252516 2392 leader_election.cc:290] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.252844 2375 raft_consensus.cc:359] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.253041 2375 raft_consensus.cc:385] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.253130 2375 raft_consensus.cc:740] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.253430 2375 consensus_queue.cc:260] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.253496 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac43df5558634550a12aa65c521d8a14" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.253726 1982 raft_consensus.cc:2468] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:21.253969 2375 ts_tablet_manager.cc:1434] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:21.254010 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac43df5558634550a12aa65c521d8a14" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:21.254292 2375 tablet_bootstrap.cc:492] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:21.254835 2195 leader_election.cc:343] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.256174 2375 tablet_bootstrap.cc:654] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.256186 2194 leader_election.cc:304] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:21.256378 2378 tablet_bootstrap.cc:654] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.256698 2392 raft_consensus.cc:2804] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:21.256816 2392 raft_consensus.cc:493] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:21.256922 2392 raft_consensus.cc:3060] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:21.259589 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac43df5558634550a12aa65c521d8a14" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:21.260156 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "604b7272209d4652989c3e6d8201a255" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:21.262415 1915 leader_election.cc:343] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.262691 2381 tablet_bootstrap.cc:492] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.262883 2381 ts_tablet_manager.cc:1403] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.149s user 0.049s sys 0.079s
I20251026 02:56:21.263899 2381 raft_consensus.cc:359] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.264101 2381 raft_consensus.cc:385] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.264248 2381 raft_consensus.cc:740] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.264510 2381 consensus_queue.cc:260] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.265259 2381 ts_tablet_manager.cc:1434] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251026 02:56:21.265584 2381 tablet_bootstrap.cc:492] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.260043 2392 raft_consensus.cc:515] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.267125 2262 raft_consensus.cc:2393] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:21.266976 2392 leader_election.cc:290] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.267493 2381 tablet_bootstrap.cc:654] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.268424 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet 20c39a62bfab429bb231ae073585d0e5 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:21.269044 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3c222bf9316b4ffe89d6899f4ded8349" } modify_external_catalogs: true
I20251026 02:56:21.269250 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3c222bf9316b4ffe89d6899f4ded8349" } modify_external_catalogs: true
I20251026 02:56:21.269996 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet b7154ef047d94a8ebc305bc3b08d108c from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:21.270478 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 7fe6cbe999c54b299d729e97f0f938e7 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:21.270961 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e56e5eec1474aefae04d278ded30765 consensus state has no leader
I20251026 02:56:21.271219 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet 3fd66abba7364951a7659011c71e76af from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:21.271668 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac43df5558634550a12aa65c521d8a14" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
W20251026 02:56:21.272634 2195 leader_election.cc:343] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.274029 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 07280105a2b74ec68e27982414b12902 (DEFAULT_TABLE table=default.Table_81ca5851f387446ead27f921baefbe9a [id=6740f80827434cc393266a57238cdbdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.274471 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07280105a2b74ec68e27982414b12902. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.274647 2375 tablet_bootstrap.cc:492] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:21.274825 2375 ts_tablet_manager.cc:1403] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.003s
I20251026 02:56:21.275735 2375 raft_consensus.cc:359] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.275992 2375 raft_consensus.cc:385] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.276073 2375 raft_consensus.cc:740] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.276373 2375 consensus_queue.cc:260] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.276881 2375 ts_tablet_manager.cc:1434] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:21.277227 2375 tablet_bootstrap.cc:492] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:21.277652 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_81ca5851f387446ead27f921baefbe9a" } modify_external_catalogs: true
I20251026 02:56:21.277885 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_81ca5851f387446ead27f921baefbe9a" } modify_external_catalogs: true
I20251026 02:56:21.279635 2375 tablet_bootstrap.cc:654] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.280342 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 5ff86fc06e7d46e78fe68cc6f31a30cc (DEFAULT_TABLE table=default.Table_d15145ea85264fc9aab94645c7ff3b41 [id=9d7e6bf7d8844942a0221a92da270f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.280336 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 07280105a2b74ec68e27982414b12902 (DEFAULT_TABLE table=default.Table_81ca5851f387446ead27f921baefbe9a [id=6740f80827434cc393266a57238cdbdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.280755 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07280105a2b74ec68e27982414b12902. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.280754 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ff86fc06e7d46e78fe68cc6f31a30cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.284476 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 3ef7247d62ec4adc82fdbed40fc852ea (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.284907 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef7247d62ec4adc82fdbed40fc852ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.284821 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 3ef7247d62ec4adc82fdbed40fc852ea (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.285315 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef7247d62ec4adc82fdbed40fc852ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.287992 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 3ef7247d62ec4adc82fdbed40fc852ea (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.288302 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 5ff86fc06e7d46e78fe68cc6f31a30cc (DEFAULT_TABLE table=default.Table_d15145ea85264fc9aab94645c7ff3b41 [id=9d7e6bf7d8844942a0221a92da270f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.288486 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef7247d62ec4adc82fdbed40fc852ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.284147 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 5ff86fc06e7d46e78fe68cc6f31a30cc (DEFAULT_TABLE table=default.Table_d15145ea85264fc9aab94645c7ff3b41 [id=9d7e6bf7d8844942a0221a92da270f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.288991 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ff86fc06e7d46e78fe68cc6f31a30cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:21.292030 2165 heartbeater.cc:646] Failed to heartbeat to 127.1.150.60:33729 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.1.150.60:33729 (error 108)
I20251026 02:56:21.292460 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7272209d4652989c3e6d8201a255 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.292754 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet ac43df5558634550a12aa65c521d8a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.293627 2376 tablet_replica.cc:333] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:21.293843 2376 raft_consensus.cc:2243] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.294025 2376 raft_consensus.cc:2272] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.295689 2376 ts_tablet_manager.cc:1916] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.287948 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 07280105a2b74ec68e27982414b12902 (DEFAULT_TABLE table=default.Table_81ca5851f387446ead27f921baefbe9a [id=6740f80827434cc393266a57238cdbdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.297305 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07280105a2b74ec68e27982414b12902. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.297807 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ff86fc06e7d46e78fe68cc6f31a30cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:21.303094 2308 heartbeater.cc:646] Failed to heartbeat to 127.1.150.60:33729 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.1.150.60:33729 (error 108)
I20251026 02:56:21.305684 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac43df5558634550a12aa65c521d8a14" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:21.305760 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b68c0b860984df99a7eeb73854a5da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.305919 1982 raft_consensus.cc:3060] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:21.306077 2372 tablet_replica.cc:333] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:21.306264 2372 raft_consensus.cc:2243] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.306476 2372 raft_consensus.cc:2272] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.307298 2372 ts_tablet_manager.cc:1916] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.309238 1982 raft_consensus.cc:2468] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:21.309765 2194 leader_election.cc:304] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:21.310222 2392 raft_consensus.cc:2804] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:21.310400 2392 raft_consensus.cc:697] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:21.310689 2392 consensus_queue.cc:237] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.316143 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "604b7272209d4652989c3e6d8201a255" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:21.316720 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac43df5558634550a12aa65c521d8a14" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:21.317605 1916 leader_election.cc:343] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.317799 1916 leader_election.cc:304] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
W20251026 02:56:21.318281 1916 raft_consensus.cc:2686] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251026 02:56:21.318849 1916 leader_election.cc:343] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.319017 1916 leader_election.cc:304] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:21.319497 2393 raft_consensus.cc:2749] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:21.325258 1915 connection.cc:537] client connection to 127.1.150.60:33729 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251026 02:56:21.325786 2027 heartbeater.cc:646] Failed to heartbeat to 127.1.150.60:33729 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251026 02:56:21.327021 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a04ac20b7624a018c403ef4f64f7a69 consensus state has no leader
I20251026 02:56:21.327456 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ff247a755b524118a9e9fcb6f8745092" } modify_external_catalogs: true
I20251026 02:56:21.327633 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ff247a755b524118a9e9fcb6f8745092" } modify_external_catalogs: true
W20251026 02:56:21.328980 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 760e6c761a7b49f695308fa1da9eda37 consensus state has no leader
I20251026 02:56:21.329340 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f26f32ab73e848fea8718bfddbebb31f" } modify_external_catalogs: true
I20251026 02:56:21.329497 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f26f32ab73e848fea8718bfddbebb31f" } modify_external_catalogs: true
W20251026 02:56:21.330698 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82315e4c5a554eeca5048a496498ca1f consensus state has no leader
I20251026 02:56:21.331081 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c550a959e98a4e22853deca526de3eae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.331625 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c550a959e98a4e22853deca526de3eae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.337590 2372 ts_tablet_manager.cc:1929] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.337769 2372 log.cc:1199] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/0b68c0b860984df99a7eeb73854a5da8
I20251026 02:56:21.338171 2372 ts_tablet_manager.cc:1950] T 0b68c0b860984df99a7eeb73854a5da8 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:21.339468 2378 tablet_bootstrap.cc:492] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.339630 2378 ts_tablet_manager.cc:1403] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.004s
I20251026 02:56:21.340615 2378 raft_consensus.cc:359] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.340862 2378 raft_consensus.cc:385] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.340970 2378 raft_consensus.cc:740] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.341984 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 0b68c0b860984df99a7eeb73854a5da8 (table default.Table_0d997fa5825e4cf2a797881f912b609f [id=319c8d881ba442c6a0629a979e01b289]) successfully deleted
I20251026 02:56:21.342428 2378 consensus_queue.cc:260] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.343122 2378 ts_tablet_manager.cc:1434] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:21.343480 2378 tablet_bootstrap.cc:492] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.345762 2378 tablet_bootstrap.cc:654] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.347887 2381 tablet_bootstrap.cc:492] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.348073 2381 ts_tablet_manager.cc:1403] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.083s user 0.015s sys 0.026s
W20251026 02:56:21.348934 1647 consensus_peers.cc:597] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 -> Peer 751b719b75ce4bc2a88a19091230ecb7 (127.1.150.60:33729): Couldn't send request to peer 751b719b75ce4bc2a88a19091230ecb7. Status: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:21.349133 2381 raft_consensus.cc:359] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:21.349371 2381 raft_consensus.cc:385] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.349459 2381 raft_consensus.cc:740] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.349748 2381 consensus_queue.cc:260] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:21.350046 2375 tablet_bootstrap.cc:492] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:21.350193 2375 ts_tablet_manager.cc:1403] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.030s
I20251026 02:56:21.350265 2381 ts_tablet_manager.cc:1434] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:21.350900 2381 tablet_bootstrap.cc:492] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.351051 2378 tablet_bootstrap.cc:492] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.351044 2375 raft_consensus.cc:359] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.351212 2375 raft_consensus.cc:385] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.351210 2378 ts_tablet_manager.cc:1403] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20251026 02:56:21.351294 2375 raft_consensus.cc:740] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.351504 2375 consensus_queue.cc:260] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.352074 2375 ts_tablet_manager.cc:1434] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:21.352418 2375 tablet_bootstrap.cc:492] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:21.352648 2381 tablet_bootstrap.cc:654] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.353701 2378 raft_consensus.cc:359] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.353946 2378 raft_consensus.cc:385] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.354035 2378 raft_consensus.cc:740] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.354352 2375 tablet_bootstrap.cc:654] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.354331 2378 consensus_queue.cc:260] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.356148 2378 ts_tablet_manager.cc:1434] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251026 02:56:21.357378 2378 tablet_bootstrap.cc:492] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.359267 2378 tablet_bootstrap.cc:654] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:21.362440 1648 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 9a04ac20b7624a018c403ef4f64f7a69 (table default.Table_0d809da9aa7f4c8cbaf054721e6aa692 [id=c162ef89d2f14e91a24a6d8c2c37c261], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9a04ac20b7624a018c403ef4f64f7a69 consensus state has no leader
I20251026 02:56:21.364233 2407 raft_consensus.cc:493] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.364439 2407 raft_consensus.cc:515] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.365644 2407 leader_election.cc:290] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:21.367085 2195 leader_election.cc:343] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 0b68c0b860984df99a7eeb73854a5da8
I20251026 02:56:21.367535 2375 tablet_bootstrap.cc:492] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
W20251026 02:56:21.367511 2194 leader_election.cc:343] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 0b68c0b860984df99a7eeb73854a5da8
I20251026 02:56:21.367631 2194 leader_election.cc:304] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.367713 2375 ts_tablet_manager.cc:1403] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20251026 02:56:21.368062 2381 tablet_bootstrap.cc:492] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.368149 2407 raft_consensus.cc:2749] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:21.368283 2381 ts_tablet_manager.cc:1403] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20251026 02:56:21.368605 2375 raft_consensus.cc:359] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.368841 2375 raft_consensus.cc:385] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.368935 2375 raft_consensus.cc:740] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.369271 2375 consensus_queue.cc:260] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.369549 2381 raft_consensus.cc:359] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.369798 2381 raft_consensus.cc:385] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.369814 2375 ts_tablet_manager.cc:1434] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:21.369911 2381 raft_consensus.cc:740] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.370221 2375 tablet_bootstrap.cc:492] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:21.370225 2381 consensus_queue.cc:260] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.371034 2381 ts_tablet_manager.cc:1434] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:21.371380 2381 tablet_bootstrap.cc:492] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.372108 2375 tablet_bootstrap.cc:654] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.373654 2381 tablet_bootstrap.cc:654] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.382838 2381 tablet_bootstrap.cc:492] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.383093 2381 ts_tablet_manager.cc:1403] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20251026 02:56:21.384327 2381 raft_consensus.cc:359] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.384630 2381 raft_consensus.cc:385] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.384728 2381 raft_consensus.cc:740] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.385036 2381 consensus_queue.cc:260] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.386673 2381 ts_tablet_manager.cc:1434] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251026 02:56:21.387133 2381 tablet_bootstrap.cc:492] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.389271 2407 raft_consensus.cc:493] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.389462 2407 raft_consensus.cc:515] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.389794 2381 tablet_bootstrap.cc:654] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.390321 2407 leader_election.cc:290] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.391209 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "760e6c761a7b49f695308fa1da9eda37" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:21.391884 2195 leader_election.cc:343] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.392910 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "760e6c761a7b49f695308fa1da9eda37" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.393239 1982 raft_consensus.cc:2468] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:21.393823 2194 leader_election.cc:304] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:21.394227 2407 raft_consensus.cc:2804] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:21.394342 2407 raft_consensus.cc:493] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:21.394429 2407 raft_consensus.cc:3060] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:21.398609 2381 tablet_bootstrap.cc:492] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.398559 2407 raft_consensus.cc:515] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.398806 2381 ts_tablet_manager.cc:1403] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20251026 02:56:21.399940 2381 raft_consensus.cc:359] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.400182 2381 raft_consensus.cc:385] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.400306 2381 raft_consensus.cc:740] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.400619 2381 consensus_queue.cc:260] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.400775 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "760e6c761a7b49f695308fa1da9eda37" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:21.401314 2381 ts_tablet_manager.cc:1434] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:21.401703 2381 tablet_bootstrap.cc:492] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:21.401742 2195 leader_election.cc:343] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.403873 2407 leader_election.cc:290] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.404510 2381 tablet_bootstrap.cc:654] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.404604 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "760e6c761a7b49f695308fa1da9eda37" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:21.404881 1982 raft_consensus.cc:3060] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:21.409286 1982 raft_consensus.cc:2468] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:21.409947 2194 leader_election.cc:304] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:21.410727 2407 raft_consensus.cc:2804] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:21.411298 2407 raft_consensus.cc:697] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:21.411792 2407 consensus_queue.cc:237] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.415035 2381 tablet_bootstrap.cc:492] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.415271 2381 ts_tablet_manager.cc:1403] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.009s
I20251026 02:56:21.416357 2381 raft_consensus.cc:359] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.416623 2381 raft_consensus.cc:385] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.416718 2381 raft_consensus.cc:740] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.417057 2381 consensus_queue.cc:260] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.417727 2381 ts_tablet_manager.cc:1434] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:21.418129 2381 tablet_bootstrap.cc:492] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.420593 2381 tablet_bootstrap.cc:654] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.423224 2405 raft_consensus.cc:493] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.423410 2405 raft_consensus.cc:515] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.424437 2405 leader_election.cc:290] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.425310 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de24439a6a3a4734a08160cb7393429a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:21.425911 2195 leader_election.cc:343] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.426345 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de24439a6a3a4734a08160cb7393429a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.426656 1982 raft_consensus.cc:2468] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:21.426728 2381 tablet_bootstrap.cc:492] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.427055 2381 ts_tablet_manager.cc:1403] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20251026 02:56:21.427202 2194 leader_election.cc:304] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:21.427570 2405 raft_consensus.cc:2804] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:21.427706 2405 raft_consensus.cc:493] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:21.427817 2405 raft_consensus.cc:3060] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:21.428517 2381 raft_consensus.cc:359] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.428895 2381 raft_consensus.cc:385] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.429072 2381 raft_consensus.cc:740] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.429468 2381 consensus_queue.cc:260] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.432099 2405 raft_consensus.cc:515] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.433845 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de24439a6a3a4734a08160cb7393429a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
W20251026 02:56:21.434914 2195 leader_election.cc:343] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.435210 2405 leader_election.cc:290] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.436460 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de24439a6a3a4734a08160cb7393429a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:21.436826 1981 raft_consensus.cc:3060] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:21.441848 1981 raft_consensus.cc:2468] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:21.442471 2194 leader_election.cc:304] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:21.443302 2440 raft_consensus.cc:2804] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:21.443519 2381 ts_tablet_manager.cc:1434] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.016s user 0.004s sys 0.011s
I20251026 02:56:21.443914 2440 raft_consensus.cc:697] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:21.445900 2381 tablet_bootstrap.cc:492] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.445652 2440 consensus_queue.cc:237] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.447801 2381 tablet_bootstrap.cc:654] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.451774 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3fd66abba7364951a7659011c71e76af
I20251026 02:56:21.452425 1656 catalog_manager.cc:6719] ReplaceTablet: tablet 3fd66abba7364951a7659011c71e76af deleted and replaced by tablet 81530bbf4a2d4e908add683475e594fb
I20251026 02:56:21.452756 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd66abba7364951a7659011c71e76af with delete_type TABLET_DATA_DELETED (replaced by tablet 81530bbf4a2d4e908add683475e594fb) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.454146 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7fe6cbe999c54b299d729e97f0f938e7
I20251026 02:56:21.454646 1663 catalog_manager.cc:6719] ReplaceTablet: tablet 7fe6cbe999c54b299d729e97f0f938e7 deleted and replaced by tablet ad55040f0a2b4cfa894d1bd058d4d69f
I20251026 02:56:21.454955 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_75c8f310e69c4237a406e4f6bd3f6b15" } modify_external_catalogs: true
I20251026 02:56:21.455139 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_75c8f310e69c4237a406e4f6bd3f6b15" } modify_external_catalogs: true
I20251026 02:56:21.455327 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fe6cbe999c54b299d729e97f0f938e7 with delete_type TABLET_DATA_DELETED (replaced by tablet ad55040f0a2b4cfa894d1bd058d4d69f) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.455578 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd66abba7364951a7659011c71e76af with delete_type TABLET_DATA_DELETED (replaced by tablet 81530bbf4a2d4e908add683475e594fb) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.455849 2377 tablet_replica.cc:333] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:21.456032 2377 raft_consensus.cc:2243] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.456245 2377 raft_consensus.cc:2272] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.455592 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fe6cbe999c54b299d729e97f0f938e7 with delete_type TABLET_DATA_DELETED (replaced by tablet ad55040f0a2b4cfa894d1bd058d4d69f) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.457062 2377 ts_tablet_manager.cc:1916] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.458209 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fe6cbe999c54b299d729e97f0f938e7 with delete_type TABLET_DATA_DELETED (replaced by tablet ad55040f0a2b4cfa894d1bd058d4d69f) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.460299 2381 tablet_bootstrap.cc:492] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.460522 2381 ts_tablet_manager.cc:1403] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.010s
W20251026 02:56:21.460626 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7fe6cbe999c54b299d729e97f0f938e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fe6cbe999c54b299d729e97f0f938e7 already in progress: creating tablet
I20251026 02:56:21.461539 2381 raft_consensus.cc:359] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.461787 2381 raft_consensus.cc:385] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.461892 2381 raft_consensus.cc:740] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.462250 2381 consensus_queue.cc:260] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.462833 2381 ts_tablet_manager.cc:1434] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:21.463428 2381 tablet_bootstrap.cc:492] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.467730 2381 tablet_bootstrap.cc:654] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.476460 2381 tablet_bootstrap.cc:492] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.476671 2381 ts_tablet_manager.cc:1403] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20251026 02:56:21.477744 2381 raft_consensus.cc:359] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.478281 2381 raft_consensus.cc:385] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.478403 2381 raft_consensus.cc:740] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.478695 2381 consensus_queue.cc:260] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.479496 2381 ts_tablet_manager.cc:1434] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251026 02:56:21.479858 2381 tablet_bootstrap.cc:492] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.480225 1662 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet ddfe9452fbe4426fb8f955b4bed586d9
I20251026 02:56:21.480731 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_673a4fc67d4f4457b78f63d90414a2d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.481523 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_673a4fc67d4f4457b78f63d90414a2d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.482728 2381 tablet_bootstrap.cc:654] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.484293 1661 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet fdfe84cd75c74d9cacd7793b09795bd3
I20251026 02:56:21.484705 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2eb5a7f69fbd436bb6e965caafc50691"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.485443 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2eb5a7f69fbd436bb6e965caafc50691 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.495922 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_995c055b47d044aeab44ffa1bb543834"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:21.500309 1663 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet efa7a5468ab8412180f1d3870fdf1632
I20251026 02:56:21.501102 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6e720c10b85f4046a9617986f013c678"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.501824 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e720c10b85f4046a9617986f013c678 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:21.502527 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_995c055b47d044aeab44ffa1bb543834 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.504359 2381 tablet_bootstrap.cc:492] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.504681 2381 ts_tablet_manager.cc:1403] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.006s
I20251026 02:56:21.505936 2381 raft_consensus.cc:359] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.506292 2381 raft_consensus.cc:385] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.506435 2381 raft_consensus.cc:740] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.506812 2381 consensus_queue.cc:260] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.507670 2381 ts_tablet_manager.cc:1434] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:21.508162 2381 tablet_bootstrap.cc:492] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.509363 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd66abba7364951a7659011c71e76af with delete_type TABLET_DATA_DELETED (replaced by tablet 81530bbf4a2d4e908add683475e594fb) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:21.510299 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3fd66abba7364951a7659011c71e76af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fd66abba7364951a7659011c71e76af already in progress: creating tablet
I20251026 02:56:21.510465 2381 tablet_bootstrap.cc:654] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.518852 2381 tablet_bootstrap.cc:492] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.519025 2381 ts_tablet_manager.cc:1403] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20251026 02:56:21.519804 2381 raft_consensus.cc:359] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.520007 2381 raft_consensus.cc:385] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.520095 2381 raft_consensus.cc:740] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.520404 2381 consensus_queue.cc:260] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.520520 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7badd06b09d543b39ca9a058c2ee80b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:21.521028 2381 ts_tablet_manager.cc:1434] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:21.521301 2381 tablet_bootstrap.cc:492] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:21.521611 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7badd06b09d543b39ca9a058c2ee80b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.523324 2381 tablet_bootstrap.cc:654] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.523699 2387 raft_consensus.cc:493] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.523973 2387 raft_consensus.cc:515] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.526544 2387 leader_election.cc:290] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:21.528461 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e071f7a8a409464d810f12e647a4ff0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:21.528767 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 (DEFAULT_TABLE table=default.Table_aa762aa5113a46a98da2f4ded4e23172 [id=2b785b2f8f2d4b0fa3f24b1ea4a66cbf]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:21.529021 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e071f7a8a409464d810f12e647a4ff0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.529752 2097 tablet_service.cc:1505] Processing CreateTablet for tablet c0f83d631faa468f828e54e97987a00b (DEFAULT_TABLE table=default.Table_50922eae951b497ba3b1e180c87facfa [id=f914c9974ac44d3f84c3d312649fe255]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.530684 2096 tablet_service.cc:1505] Processing CreateTablet for tablet b108f0d3dae84fab93bff012400c4a2a (DEFAULT_TABLE table=default.Table_17277a07df7c41a3a3b95c89e0459a5b [id=2f5d9104d21c49e0985a01ffb4b7a978]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.531783 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 (DEFAULT_TABLE table=default.Table_aa762aa5113a46a98da2f4ded4e23172 [id=2b785b2f8f2d4b0fa3f24b1ea4a66cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.531973 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 565255802f4f42ba8320a3ec80d2db8f (DEFAULT_TABLE table=default.Table_cf4652bce4e447caa066bfebc7b43e50 [id=1f46f0e30c4045419969ccc51df359f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.532516 1953 tablet_service.cc:1505] Processing CreateTablet for tablet c0f83d631faa468f828e54e97987a00b (DEFAULT_TABLE table=default.Table_50922eae951b497ba3b1e180c87facfa [id=f914c9974ac44d3f84c3d312649fe255]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.530114 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.532912 1962 tablet_service.cc:1505] Processing CreateTablet for tablet b108f0d3dae84fab93bff012400c4a2a (DEFAULT_TABLE table=default.Table_17277a07df7c41a3a3b95c89e0459a5b [id=2f5d9104d21c49e0985a01ffb4b7a978]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.533003 2099 tablet_service.cc:1505] Processing CreateTablet for tablet a2bfbe624075463f8b90ed4ec66b784c (DEFAULT_TABLE table=default.Table_03c78b21a07841b2b1a8962abe80a10e [id=8a3d1c20763445c499bd655eb99deb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.533217 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.533265 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 565255802f4f42ba8320a3ec80d2db8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.533475 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bfbe624075463f8b90ed4ec66b784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.535260 2098 tablet_service.cc:1505] Processing CreateTablet for tablet e0101098df05462bb26b3c187b259c08 (DEFAULT_TABLE table=default.Table_bcf0aac1b41747e9a94a97b8430fa081 [id=df8666bd175f43ceb47649f67a3d865e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.535640 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0101098df05462bb26b3c187b259c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.536351 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f83d631faa468f828e54e97987a00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.536844 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b108f0d3dae84fab93bff012400c4a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.540362 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b108f0d3dae84fab93bff012400c4a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.540583 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f83d631faa468f828e54e97987a00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.545675 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 (DEFAULT_TABLE table=default.Table_aa762aa5113a46a98da2f4ded4e23172 [id=2b785b2f8f2d4b0fa3f24b1ea4a66cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.546108 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.546749 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 2487bf7d40f34922aa2fd165488342f6 (DEFAULT_TABLE table=default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.547101 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2487bf7d40f34922aa2fd165488342f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.547946 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 0e89c69e3b58458aad10beaeb5128d60 (DEFAULT_TABLE table=default.Table_ad061592630140d9952cf22072721a4d [id=347d712aca4e477c89805846800fb824]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.548357 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e89c69e3b58458aad10beaeb5128d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.548501 2240 tablet_service.cc:1505] Processing CreateTablet for tablet c0f83d631faa468f828e54e97987a00b (DEFAULT_TABLE table=default.Table_50922eae951b497ba3b1e180c87facfa [id=f914c9974ac44d3f84c3d312649fe255]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.548900 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f83d631faa468f828e54e97987a00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.545701 2237 tablet_service.cc:1505] Processing CreateTablet for tablet b108f0d3dae84fab93bff012400c4a2a (DEFAULT_TABLE table=default.Table_17277a07df7c41a3a3b95c89e0459a5b [id=2f5d9104d21c49e0985a01ffb4b7a978]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.552704 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b108f0d3dae84fab93bff012400c4a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.556161 2440 raft_consensus.cc:493] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.556358 2440 raft_consensus.cc:515] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.557035 2440 leader_election.cc:290] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.558566 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 565255802f4f42ba8320a3ec80d2db8f (DEFAULT_TABLE table=default.Table_cf4652bce4e447caa066bfebc7b43e50 [id=1f46f0e30c4045419969ccc51df359f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.559019 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 565255802f4f42ba8320a3ec80d2db8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.559056 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 565255802f4f42ba8320a3ec80d2db8f (DEFAULT_TABLE table=default.Table_cf4652bce4e447caa066bfebc7b43e50 [id=1f46f0e30c4045419969ccc51df359f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.559515 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 565255802f4f42ba8320a3ec80d2db8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.560562 2243 tablet_service.cc:1505] Processing CreateTablet for tablet a2bfbe624075463f8b90ed4ec66b784c (DEFAULT_TABLE table=default.Table_03c78b21a07841b2b1a8962abe80a10e [id=8a3d1c20763445c499bd655eb99deb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.560971 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bfbe624075463f8b90ed4ec66b784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.563639 2235 tablet_service.cc:1505] Processing CreateTablet for tablet e0101098df05462bb26b3c187b259c08 (DEFAULT_TABLE table=default.Table_bcf0aac1b41747e9a94a97b8430fa081 [id=df8666bd175f43ceb47649f67a3d865e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.564105 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0101098df05462bb26b3c187b259c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.564543 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 2487bf7d40f34922aa2fd165488342f6 (DEFAULT_TABLE table=default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.564532 1960 tablet_service.cc:1505] Processing CreateTablet for tablet a2bfbe624075463f8b90ed4ec66b784c (DEFAULT_TABLE table=default.Table_03c78b21a07841b2b1a8962abe80a10e [id=8a3d1c20763445c499bd655eb99deb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.564968 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2487bf7d40f34922aa2fd165488342f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.565027 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bfbe624075463f8b90ed4ec66b784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.568645 1961 tablet_service.cc:1505] Processing CreateTablet for tablet e0101098df05462bb26b3c187b259c08 (DEFAULT_TABLE table=default.Table_bcf0aac1b41747e9a94a97b8430fa081 [id=df8666bd175f43ceb47649f67a3d865e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.569146 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0101098df05462bb26b3c187b259c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.572361 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3802bec60e074107b9a8f8ddc3895fe2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:21.572724 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 2487bf7d40f34922aa2fd165488342f6 (DEFAULT_TABLE table=default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.573122 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2487bf7d40f34922aa2fd165488342f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.577107 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5edbe48bb8ce41de84eaeaa4b4137fdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.577940 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5edbe48bb8ce41de84eaeaa4b4137fdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.584002 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_797feb703f214aac9ea3dccac59e202a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.584838 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_797feb703f214aac9ea3dccac59e202a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:21.594012 2195 leader_election.cc:343] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.598616 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 0e89c69e3b58458aad10beaeb5128d60 (DEFAULT_TABLE table=default.Table_ad061592630140d9952cf22072721a4d [id=347d712aca4e477c89805846800fb824]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.598660 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 0e89c69e3b58458aad10beaeb5128d60 (DEFAULT_TABLE table=default.Table_ad061592630140d9952cf22072721a4d [id=347d712aca4e477c89805846800fb824]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.599469 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e89c69e3b58458aad10beaeb5128d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.610381 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8c6fbc31d0674b8ebb6d80c95431b26f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.611155 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c6fbc31d0674b8ebb6d80c95431b26f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.614318 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0d809da9aa7f4c8cbaf054721e6aa692" } new_table_name: "default.Table_627cb33ef7ce4082b858fa0670d09862" modify_external_catalogs: true
I20251026 02:56:21.621181 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e89c69e3b58458aad10beaeb5128d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.630151 2377 ts_tablet_manager.cc:1929] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.630506 2377 log.cc:1199] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3fd66abba7364951a7659011c71e76af
I20251026 02:56:21.631125 2377 ts_tablet_manager.cc:1950] T 3fd66abba7364951a7659011c71e76af P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:21.632055 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3fd66abba7364951a7659011c71e76af (table default.Table_99b42c4a4a874b68bb4beba4c08cdeb0 [id=472d44e7e6f0409ab9d238c4913cc8a8]) successfully deleted
I20251026 02:56:21.669152 2381 tablet_bootstrap.cc:492] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.669363 2381 ts_tablet_manager.cc:1403] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.148s user 0.003s sys 0.001s
I20251026 02:56:21.670234 2381 raft_consensus.cc:359] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.670440 2381 raft_consensus.cc:385] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.670503 2381 raft_consensus.cc:740] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.670781 2381 consensus_queue.cc:260] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.671633 2381 ts_tablet_manager.cc:1434] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:21.671962 2381 tablet_bootstrap.cc:492] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.674093 2381 tablet_bootstrap.cc:654] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.679878 2381 tablet_bootstrap.cc:492] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.680059 2381 ts_tablet_manager.cc:1403] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20251026 02:56:21.681870 2381 raft_consensus.cc:359] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.682091 2381 raft_consensus.cc:385] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.682186 2381 raft_consensus.cc:740] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.683697 2381 consensus_queue.cc:260] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.684721 2381 ts_tablet_manager.cc:1434] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251026 02:56:21.685048 2381 tablet_bootstrap.cc:492] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.686671 2440 raft_consensus.cc:493] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.686915 2440 raft_consensus.cc:515] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.687862 2440 leader_election.cc:290] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.688792 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a4ba66147c249728713a42d20f541f4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:21.688946 2381 tablet_bootstrap.cc:654] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.689075 2119 raft_consensus.cc:2468] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:21.689654 2195 leader_election.cc:304] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:21.690116 2440 raft_consensus.cc:2804] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:21.690279 2440 raft_consensus.cc:493] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:21.690409 2440 raft_consensus.cc:3060] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:21.691434 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3802bec60e074107b9a8f8ddc3895fe2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:21.692551 2194 leader_election.cc:343] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 3a4ba66147c249728713a42d20f541f4
W20251026 02:56:21.692987 2194 leader_election.cc:343] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.693132 2194 leader_election.cc:304] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.694396 2440 raft_consensus.cc:515] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.694594 2392 raft_consensus.cc:2749] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:21.695781 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a4ba66147c249728713a42d20f541f4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:21.695884 2381 tablet_bootstrap.cc:492] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.696003 2119 raft_consensus.cc:3060] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:21.696072 2381 ts_tablet_manager.cc:1403] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20251026 02:56:21.700066 2119 raft_consensus.cc:2468] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:21.700771 2195 leader_election.cc:304] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:21.700814 2381 raft_consensus.cc:359] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.701023 2381 raft_consensus.cc:385] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.701118 2381 raft_consensus.cc:740] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.701287 2440 leader_election.cc:290] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.701562 2440 raft_consensus.cc:493] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.701833 2440 raft_consensus.cc:515] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.701962 2381 consensus_queue.cc:260] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.702656 2440 leader_election.cc:290] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.703294 2440 raft_consensus.cc:2804] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:21.703590 2440 raft_consensus.cc:697] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:21.704169 2440 consensus_queue.cc:237] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.705062 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 07280105a2b74ec68e27982414b12902
I20251026 02:56:21.705191 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20c39a62bfab429bb231ae073585d0e5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:21.705749 2381 ts_tablet_manager.cc:1434] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
W20251026 02:56:21.705766 2195 leader_election.cc:343] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.705852 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet 3ef7247d62ec4adc82fdbed40fc852ea from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:21.706107 2381 tablet_bootstrap.cc:492] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.706864 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 07280105a2b74ec68e27982414b12902 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.706835 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 07280105a2b74ec68e27982414b12902 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:21.707587 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 07280105a2b74ec68e27982414b12902 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07280105a2b74ec68e27982414b12902 already in progress: creating tablet
I20251026 02:56:21.707991 2381 tablet_bootstrap.cc:654] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.713801 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20c39a62bfab429bb231ae073585d0e5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.714951 2393 raft_consensus.cc:493] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.715245 2393 raft_consensus.cc:515] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.715341 2405 raft_consensus.cc:493] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.715484 2405 raft_consensus.cc:515] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.716054 2393 leader_election.cc:290] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:21.716722 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82315e4c5a554eeca5048a496498ca1f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:21.717295 1916 leader_election.cc:343] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.717823 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82315e4c5a554eeca5048a496498ca1f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:21.718045 2262 raft_consensus.cc:2468] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:21.718123 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 07280105a2b74ec68e27982414b12902 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.718797 1915 leader_election.cc:304] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:21.719297 2393 raft_consensus.cc:2804] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:21.719396 2393 raft_consensus.cc:493] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:21.719493 2393 raft_consensus.cc:3060] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:21.720134 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:21.720395 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: This master's current role is: LEADER
I20251026 02:56:21.722780 2393 raft_consensus.cc:515] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.724174 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82315e4c5a554eeca5048a496498ca1f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:21.724746 2378 tablet_bootstrap.cc:492] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
W20251026 02:56:21.724826 1916 leader_election.cc:343] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.724918 2378 ts_tablet_manager.cc:1403] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.368s user 0.079s sys 0.178s
I20251026 02:56:21.725800 2378 raft_consensus.cc:359] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.725981 2378 raft_consensus.cc:385] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.726074 2378 raft_consensus.cc:740] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.726079 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82315e4c5a554eeca5048a496498ca1f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:21.726269 2262 raft_consensus.cc:3060] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:21.726363 2378 consensus_queue.cc:260] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.728507 2393 leader_election.cc:290] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:21.730779 2262 raft_consensus.cc:2468] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:21.731524 1915 leader_election.cc:304] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:21.731776 2376 ts_tablet_manager.cc:1929] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.731998 2376 log.cc:1199] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/604b7272209d4652989c3e6d8201a255
I20251026 02:56:21.732465 2376 ts_tablet_manager.cc:1950] T 604b7272209d4652989c3e6d8201a255 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:21.732599 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82315e4c5a554eeca5048a496498ca1f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:21.733337 2195 leader_election.cc:343] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.733738 2376 tablet_replica.cc:333] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:21.734025 2376 raft_consensus.cc:2243] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.734257 2376 raft_consensus.cc:2272] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.734869 2376 ts_tablet_manager.cc:1916] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.736259 2393 raft_consensus.cc:2804] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:21.736487 2393 raft_consensus.cc:697] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:21.736672 2405 leader_election.cc:290] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.737773 2393 consensus_queue.cc:237] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:21.741482 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7fe6cbe999c54b299d729e97f0f938e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fe6cbe999c54b299d729e97f0f938e7 already in progress: creating tablet
W20251026 02:56:21.742262 2194 leader_election.cc:343] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 3a4ba66147c249728713a42d20f541f4
W20251026 02:56:21.742739 2194 leader_election.cc:343] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.744292 2194 leader_election.cc:304] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.744663 2440 raft_consensus.cc:2749] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:21.745913 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 07280105a2b74ec68e27982414b12902 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07280105a2b74ec68e27982414b12902 already in progress: creating tablet
I20251026 02:56:21.748783 2381 tablet_bootstrap.cc:492] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.748962 2381 ts_tablet_manager.cc:1403] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20251026 02:56:21.749938 2381 raft_consensus.cc:359] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.750192 2381 raft_consensus.cc:385] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.750301 2381 raft_consensus.cc:740] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.750543 2381 consensus_queue.cc:260] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.754150 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 604b7272209d4652989c3e6d8201a255 (table default.Table_e00202a12c9349eda8d47f735e4cddab [id=d7ac91b3b98441baaa6ecc48cb904c7e]) successfully deleted
I20251026 02:56:21.754180 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82315e4c5a554eeca5048a496498ca1f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.754475 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0d066a461d9e4df198968c2e6f11ae2d
I20251026 02:56:21.755076 1655 catalog_manager.cc:6719] ReplaceTablet: tablet 0d066a461d9e4df198968c2e6f11ae2d deleted and replaced by tablet 20dec74440cf4bb9949c462ec51e9889
I20251026 02:56:21.756686 2194 leader_election.cc:304] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.756829 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet de24439a6a3a4734a08160cb7393429a
I20251026 02:56:21.756942 2440 raft_consensus.cc:2749] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:21.757095 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20dec74440cf4bb9949c462ec51e9889 has no consensus state
I20251026 02:56:21.757364 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 3fd66abba7364951a7659011c71e76af from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:21.759752 2381 ts_tablet_manager.cc:1434] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20251026 02:56:21.760100 2381 tablet_bootstrap.cc:492] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.760574 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet de24439a6a3a4734a08160cb7393429a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.760665 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d066a461d9e4df198968c2e6f11ae2d with delete_type TABLET_DATA_DELETED (replaced by tablet 20dec74440cf4bb9949c462ec51e9889) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.761181 2372 tablet_replica.cc:333] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:21.761360 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet de24439a6a3a4734a08160cb7393429a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de24439a6a3a4734a08160cb7393429a already in progress: creating tablet
I20251026 02:56:21.761384 2372 raft_consensus.cc:2243] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.761581 2372 raft_consensus.cc:2272] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.761956 2381 tablet_bootstrap.cc:654] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.762174 2372 ts_tablet_manager.cc:1916] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.763173 2378 ts_tablet_manager.cc:1434] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.038s user 0.004s sys 0.001s
I20251026 02:56:21.763499 2378 tablet_bootstrap.cc:492] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.765285 2378 tablet_bootstrap.cc:654] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.765996 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d066a461d9e4df198968c2e6f11ae2d with delete_type TABLET_DATA_DELETED (replaced by tablet 20dec74440cf4bb9949c462ec51e9889) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.767306 2405 raft_consensus.cc:493] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.767470 2405 raft_consensus.cc:515] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.767585 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet de24439a6a3a4734a08160cb7393429a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.768115 2405 leader_election.cc:290] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.768286 2405 raft_consensus.cc:493] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.768390 2405 raft_consensus.cc:515] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.768885 2405 leader_election.cc:290] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.771281 2377 tablet_replica.cc:333] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:21.773737 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19edc0546fd04110b37353178da2652e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.775424 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19edc0546fd04110b37353178da2652e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:21.780980 2372 ts_tablet_manager.cc:1929] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.781265 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet de24439a6a3a4734a08160cb7393429a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.781626 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d066a461d9e4df198968c2e6f11ae2d with delete_type TABLET_DATA_DELETED (replaced by tablet 20dec74440cf4bb9949c462ec51e9889) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.783013 2372 log.cc:1199] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/0d066a461d9e4df198968c2e6f11ae2d
I20251026 02:56:21.783047 2378 tablet_bootstrap.cc:492] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.783221 2378 ts_tablet_manager.cc:1403] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20251026 02:56:21.783622 2372 ts_tablet_manager.cc:1950] T 0d066a461d9e4df198968c2e6f11ae2d P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:21.784109 2378 raft_consensus.cc:359] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.784333 2378 raft_consensus.cc:385] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251026 02:56:21.776046 2194 leader_election.cc:343] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.784413 2378 raft_consensus.cc:740] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.784613 2377 raft_consensus.cc:2243] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:21.784695 2378 consensus_queue.cc:260] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.785403 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 0d066a461d9e4df198968c2e6f11ae2d (table default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]) successfully deleted
I20251026 02:56:21.786491 2378 ts_tablet_manager.cc:1434] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251026 02:56:21.786847 2378 tablet_bootstrap.cc:492] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.788726 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_03c78b21a07841b2b1a8962abe80a10e" } new_table_name: "default.Table_54ffd2743e604aa4bb1a694ca3217891" modify_external_catalogs: true
I20251026 02:56:21.789815 2378 tablet_bootstrap.cc:654] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.792172 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1854fa4e2e14427aadd889784db2cc44" } new_table_name: "default.Table_6b07e3e332c141aa824ea90a76f40208" modify_external_catalogs: true
I20251026 02:56:21.792488 1658 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 760e6c761a7b49f695308fa1da9eda37 (table default.Table_1854fa4e2e14427aadd889784db2cc44 [id=c3e9c585b810403f88e4eb2b4924b73c]). Expected version 1 got 0
I20251026 02:56:21.792936 1658 catalog_manager.cc:5649] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1). New cstate: current_term: 1 leader_uuid: "7fef01f4f2b843538ea245a09dd087a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:21.793751 2378 tablet_bootstrap.cc:492] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.793897 2378 ts_tablet_manager.cc:1403] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20251026 02:56:21.794348 1658 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 82315e4c5a554eeca5048a496498ca1f (table default.Table_cfeac8daaf08492a9456d2e06cd62a58 [id=1d09f5b9fa164ac783f0be9332eac4c3]). Expected version 2 got 0
I20251026 02:56:21.794402 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1466d8b694c74b75bbc0b928e917993a" } new_table_name: "default.Table_0da51595453549b3b7e077f7b7ff3ec7" modify_external_catalogs: true
I20251026 02:56:21.794765 2378 raft_consensus.cc:359] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.794952 2378 raft_consensus.cc:385] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.795050 2378 raft_consensus.cc:740] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.795428 2378 consensus_queue.cc:260] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.795954 2378 ts_tablet_manager.cc:1434] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:21.797274 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 652112d5039943628550188522bf0a17 with delete_type TABLET_DATA_DELETED (replaced by tablet 701f13f90f154e378b353e84d1bebd4e) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:21.776166 2195 leader_election.cc:343] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 3f4436f64997445aa880610c5a93f955
I20251026 02:56:21.797626 2372 tablet_replica.cc:333] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:21.797813 2372 raft_consensus.cc:2243] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:21.797861 2195 leader_election.cc:343] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.798028 2372 raft_consensus.cc:2272] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.798005 2195 leader_election.cc:304] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.798072 2377 pending_rounds.cc:70] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:21.798308 2377 pending_rounds.cc:77] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888472777658368 op_type: NO_OP noop_request { }
I20251026 02:56:21.798393 2405 raft_consensus.cc:2749] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:21.798529 2377 raft_consensus.cc:2889] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:21.798588 2372 ts_tablet_manager.cc:1916] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:21.798594 2194 leader_election.cc:343] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 3f4436f64997445aa880610c5a93f955
I20251026 02:56:21.798676 2377 raft_consensus.cc:2272] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.798976 2194 leader_election.cc:304] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.798535 2440 raft_consensus.cc:493] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.799448 2440 raft_consensus.cc:515] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:21.799748 2377 ts_tablet_manager.cc:1916] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.800042 2440 leader_election.cc:290] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:21.800189 2440 raft_consensus.cc:2749] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:21.800865 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "604b7272209d4652989c3e6d8201a255" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:21.801237 2194 leader_election.cc:343] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 604b7272209d4652989c3e6d8201a255
I20251026 02:56:21.791560 2381 tablet_bootstrap.cc:492] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.801503 2381 ts_tablet_manager.cc:1403] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20251026 02:56:21.802268 2381 raft_consensus.cc:359] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.802472 2381 raft_consensus.cc:385] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.802563 2381 raft_consensus.cc:740] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
W20251026 02:56:21.802646 2195 leader_election.cc:343] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.802783 2195 leader_election.cc:304] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.802829 2381 consensus_queue.cc:260] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.803105 2440 raft_consensus.cc:2749] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:21.803544 2381 ts_tablet_manager.cc:1434] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:21.803889 2381 tablet_bootstrap.cc:492] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.804894 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac43df5558634550a12aa65c521d8a14" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.805667 2381 tablet_bootstrap.cc:654] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.805800 2372 ts_tablet_manager.cc:1929] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.806013 2372 log.cc:1199] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/652112d5039943628550188522bf0a17
I20251026 02:56:21.806857 2372 ts_tablet_manager.cc:1950] T 652112d5039943628550188522bf0a17 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:21.806833 2053 leader_election.cc:343] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: Tablet not RUNNING: STOPPED
I20251026 02:56:21.807284 2378 tablet_bootstrap.cc:492] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.808145 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 652112d5039943628550188522bf0a17 (table default.Table_233cc7c9cac543889a30f7a5a8d22156 [id=01000ef9384644bdb76c6cba086b605d]) successfully deleted
I20251026 02:56:21.808974 2378 tablet_bootstrap.cc:654] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.809252 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac43df5558634550a12aa65c521d8a14" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:21.809918 2381 tablet_bootstrap.cc:492] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.810091 2381 ts_tablet_manager.cc:1403] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251026 02:56:21.810709 2053 leader_election.cc:304] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.811165 2387 raft_consensus.cc:2749] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:21.811769 2381 raft_consensus.cc:359] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.811961 2381 raft_consensus.cc:385] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.812057 2381 raft_consensus.cc:740] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.813356 2381 consensus_queue.cc:260] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.813827 2381 ts_tablet_manager.cc:1434] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251026 02:56:21.814105 2381 tablet_bootstrap.cc:492] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.817229 2375 tablet_bootstrap.cc:492] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:21.817392 2375 ts_tablet_manager.cc:1403] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.447s user 0.165s sys 0.197s
I20251026 02:56:21.818222 2375 raft_consensus.cc:359] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:21.818409 2375 raft_consensus.cc:385] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.818511 2375 raft_consensus.cc:740] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.818634 2405 raft_consensus.cc:493] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.818747 2375 consensus_queue.cc:260] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:21.818766 2405 raft_consensus.cc:515] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.819240 2375 ts_tablet_manager.cc:1434] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:21.819409 2405 leader_election.cc:290] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.820111 2375 tablet_bootstrap.cc:492] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:21.820286 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eea9b18e5094a22a51942f438a20eb4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:21.820894 2195 leader_election.cc:343] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.821964 2375 tablet_bootstrap.cc:654] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.822130 2378 tablet_bootstrap.cc:492] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.822342 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eea9b18e5094a22a51942f438a20eb4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.822746 2381 tablet_bootstrap.cc:654] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:21.822904 2194 leader_election.cc:343] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.823078 2194 leader_election.cc:304] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.823504 2405 raft_consensus.cc:2749] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:21.823809 2378 ts_tablet_manager.cc:1403] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.006s
I20251026 02:56:21.824622 2378 raft_consensus.cc:359] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.824827 2378 raft_consensus.cc:385] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.824918 2378 raft_consensus.cc:740] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.825249 2378 consensus_queue.cc:260] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.825767 2381 tablet_bootstrap.cc:492] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.825816 2378 ts_tablet_manager.cc:1434] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:21.825928 2381 ts_tablet_manager.cc:1403] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20251026 02:56:21.826122 2378 tablet_bootstrap.cc:492] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.826714 2381 raft_consensus.cc:359] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.826892 2381 raft_consensus.cc:385] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.826958 2381 raft_consensus.cc:740] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.827234 2381 consensus_queue.cc:260] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.827829 2381 ts_tablet_manager.cc:1434] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:21.828058 2378 tablet_bootstrap.cc:654] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.828114 2381 tablet_bootstrap.cc:492] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.829926 2381 tablet_bootstrap.cc:654] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.833870 2375 tablet_bootstrap.cc:492] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:21.834054 2375 ts_tablet_manager.cc:1403] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20251026 02:56:21.835074 2375 raft_consensus.cc:359] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.835294 2375 raft_consensus.cc:385] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.835364 2375 raft_consensus.cc:740] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.835585 2375 consensus_queue.cc:260] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.836099 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet 5e56e5eec1474aefae04d278ded30765 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:21.837304 2375 ts_tablet_manager.cc:1434] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 02:56:21.837654 2375 tablet_bootstrap.cc:492] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:21.839282 2375 tablet_bootstrap.cc:654] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.842120 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet 604b7272209d4652989c3e6d8201a255 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:21.845861 2378 tablet_bootstrap.cc:492] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.846038 2378 ts_tablet_manager.cc:1403] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.005s
I20251026 02:56:21.846876 2378 raft_consensus.cc:359] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.847095 2378 raft_consensus.cc:385] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.847215 2378 raft_consensus.cc:740] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.847501 2378 consensus_queue.cc:260] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.850934 2376 ts_tablet_manager.cc:1929] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.851207 2376 log.cc:1199] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/ac43df5558634550a12aa65c521d8a14
I20251026 02:56:21.851675 2376 ts_tablet_manager.cc:1950] T ac43df5558634550a12aa65c521d8a14 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:21.851778 2378 ts_tablet_manager.cc:1434] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251026 02:56:21.853111 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet ac43df5558634550a12aa65c521d8a14 (table default.Table_8827f142daf64c45b7786c00d72ace16 [id=c881f3e0b92f45788e52269792793f47]) successfully deleted
I20251026 02:56:21.853511 2376 tablet_replica.cc:333] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:21.853768 2376 raft_consensus.cc:2243] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.854022 2376 raft_consensus.cc:2272] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.854837 2376 ts_tablet_manager.cc:1916] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.855232 2378 tablet_bootstrap.cc:492] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.855340 2405 raft_consensus.cc:493] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.855535 2405 raft_consensus.cc:515] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.856316 2405 leader_election.cc:290] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.857000 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "426ce6fb80874df68210a652f7b7602f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.857379 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "426ce6fb80874df68210a652f7b7602f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:21.857582 2194 leader_election.cc:343] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.857764 2378 tablet_bootstrap.cc:654] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:21.857975 2195 leader_election.cc:343] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.858103 2195 leader_election.cc:304] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.858493 2405 raft_consensus.cc:2749] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:21.862293 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3ef7247d62ec4adc82fdbed40fc852ea
I20251026 02:56:21.862872 1660 catalog_manager.cc:6719] ReplaceTablet: tablet 3ef7247d62ec4adc82fdbed40fc852ea deleted and replaced by tablet 0c647782049243d6bcd595f1a10d89c6
I20251026 02:56:21.863204 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_995c055b47d044aeab44ffa1bb543834" } modify_external_catalogs: true
I20251026 02:56:21.863397 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_995c055b47d044aeab44ffa1bb543834" } modify_external_catalogs: true
I20251026 02:56:21.864135 2376 ts_tablet_manager.cc:1929] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.864317 2376 log.cc:1199] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3fd66abba7364951a7659011c71e76af
I20251026 02:56:21.864738 2376 ts_tablet_manager.cc:1950] T 3fd66abba7364951a7659011c71e76af P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:21.864881 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef7247d62ec4adc82fdbed40fc852ea with delete_type TABLET_DATA_DELETED (replaced by tablet 0c647782049243d6bcd595f1a10d89c6) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.865564 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef7247d62ec4adc82fdbed40fc852ea with delete_type TABLET_DATA_DELETED (replaced by tablet 0c647782049243d6bcd595f1a10d89c6) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.865857 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3fd66abba7364951a7659011c71e76af (table default.Table_99b42c4a4a874b68bb4beba4c08cdeb0 [id=472d44e7e6f0409ab9d238c4913cc8a8]) successfully deleted
W20251026 02:56:21.866235 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7fe6cbe999c54b299d729e97f0f938e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fe6cbe999c54b299d729e97f0f938e7 already in progress: creating tablet
I20251026 02:56:21.866418 2376 tablet_replica.cc:333] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
W20251026 02:56:21.866511 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 07280105a2b74ec68e27982414b12902 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07280105a2b74ec68e27982414b12902 already in progress: creating tablet
W20251026 02:56:21.866545 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3ef7247d62ec4adc82fdbed40fc852ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef7247d62ec4adc82fdbed40fc852ea already in progress: creating tablet
I20251026 02:56:21.866662 2376 raft_consensus.cc:2243] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.866838 2376 raft_consensus.cc:2272] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.867028 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef7247d62ec4adc82fdbed40fc852ea with delete_type TABLET_DATA_DELETED (replaced by tablet 0c647782049243d6bcd595f1a10d89c6) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.867414 2376 ts_tablet_manager.cc:1916] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.870247 2378 tablet_bootstrap.cc:492] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.870409 2378 ts_tablet_manager.cc:1403] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20251026 02:56:21.871012 2378 raft_consensus.cc:359] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.871176 2378 raft_consensus.cc:385] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.871261 2378 raft_consensus.cc:740] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.871490 2378 consensus_queue.cc:260] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.871892 2378 ts_tablet_manager.cc:1434] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20251026 02:56:21.872255 2378 tablet_bootstrap.cc:492] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.874046 2378 tablet_bootstrap.cc:654] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.879669 2376 ts_tablet_manager.cc:1929] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.879796 2376 log.cc:1199] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/de24439a6a3a4734a08160cb7393429a
I20251026 02:56:21.880136 2376 ts_tablet_manager.cc:1950] T de24439a6a3a4734a08160cb7393429a P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
W20251026 02:56:21.882613 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2bfbe624075463f8b90ed4ec66b784c consensus state has no leader
I20251026 02:56:21.884373 2377 ts_tablet_manager.cc:1929] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:21.884537 2377 log.cc:1199] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/0d066a461d9e4df198968c2e6f11ae2d
I20251026 02:56:21.885012 2377 ts_tablet_manager.cc:1950] T 0d066a461d9e4df198968c2e6f11ae2d P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:21.884918 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c550a959e98a4e22853deca526de3eae" } modify_external_catalogs: true
I20251026 02:56:21.885152 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c550a959e98a4e22853deca526de3eae" } modify_external_catalogs: true
I20251026 02:56:21.886300 2377 tablet_replica.cc:333] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:21.886425 2376 tablet_replica.cc:333] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:21.886581 2377 raft_consensus.cc:2243] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:21.886883 2377 pending_rounds.cc:70] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:21.886896 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 0d066a461d9e4df198968c2e6f11ae2d (table default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]) successfully deleted
I20251026 02:56:21.886940 2376 raft_consensus.cc:2243] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.886976 2377 pending_rounds.cc:77] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888474405642240 op_type: NO_OP noop_request { }
I20251026 02:56:21.887095 2376 raft_consensus.cc:2272] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.887105 2377 raft_consensus.cc:2889] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:21.887176 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet de24439a6a3a4734a08160cb7393429a (table default.Table_3c222bf9316b4ffe89d6899f4ded8349 [id=931430a4a03741bd86175eabf9764a77]) successfully deleted
I20251026 02:56:21.887236 2377 raft_consensus.cc:2272] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.888528 2376 ts_tablet_manager.cc:1916] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.893772 2381 tablet_bootstrap.cc:492] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:21.893852 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 9a04ac20b7624a018c403ef4f64f7a69 (table default.Table_0d809da9aa7f4c8cbaf054721e6aa692 [id=c162ef89d2f14e91a24a6d8c2c37c261]). Expected version 1 got 0
I20251026 02:56:21.893946 2381 ts_tablet_manager.cc:1403] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.024s
I20251026 02:56:21.894030 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (table default.Table_537013d780884ee89ee2ad679f506a49 [id=095ad208529348d4aea2d4e2840e07c7]). Expected version 2 got 0
I20251026 02:56:21.894181 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 5e56e5eec1474aefae04d278ded30765 (table default.Table_83bdf2cfef4042748553ebe8ae34996a [id=6734e5731cc94e71b43c76512f649d57]). Expected version 1 got 0
I20251026 02:56:21.894407 1655 catalog_manager.cc:5649] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:21.894858 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 760e6c761a7b49f695308fa1da9eda37 (table default.Table_1854fa4e2e14427aadd889784db2cc44 [id=c3e9c585b810403f88e4eb2b4924b73c]). Expected version 1 got 0
I20251026 02:56:21.895027 2376 ts_tablet_manager.cc:1929] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.895154 2376 log.cc:1199] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/0d066a461d9e4df198968c2e6f11ae2d
I20251026 02:56:21.895165 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 82315e4c5a554eeca5048a496498ca1f (table default.Table_cfeac8daaf08492a9456d2e06cd62a58 [id=1d09f5b9fa164ac783f0be9332eac4c3]). Expected version 2 got 0
I20251026 02:56:21.895066 2381 raft_consensus.cc:359] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.895313 2381 raft_consensus.cc:385] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.895421 2381 raft_consensus.cc:740] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.895506 2376 ts_tablet_manager.cc:1950] T 0d066a461d9e4df198968c2e6f11ae2d P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
W20251026 02:56:21.895643 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a04ac20b7624a018c403ef4f64f7a69 consensus state has no leader
I20251026 02:56:21.895706 2381 consensus_queue.cc:260] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.896155 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_cf81dba46e2b4f6da4bd597901aaddcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.896802 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf81dba46e2b4f6da4bd597901aaddcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.897688 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cf4652bce4e447caa066bfebc7b43e50" } modify_external_catalogs: true
I20251026 02:56:21.897959 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cf4652bce4e447caa066bfebc7b43e50" } modify_external_catalogs: true
I20251026 02:56:21.899232 2378 tablet_bootstrap.cc:492] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.899366 2378 ts_tablet_manager.cc:1403] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.004s
W20251026 02:56:21.899474 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 760e6c761a7b49f695308fa1da9eda37 consensus state has no leader
I20251026 02:56:21.900169 2378 raft_consensus.cc:359] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.900357 2378 raft_consensus.cc:385] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.900446 2378 raft_consensus.cc:740] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.900715 2378 consensus_queue.cc:260] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.901341 2378 ts_tablet_manager.cc:1434] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:21.901620 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 0d066a461d9e4df198968c2e6f11ae2d (table default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]) successfully deleted
I20251026 02:56:21.901892 2378 tablet_bootstrap.cc:492] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:21.902138 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 64d00ae95e14439eaa748d17ca67118a with delete_type TABLET_DATA_DELETED (replaced by tablet 3ef7247d62ec4adc82fdbed40fc852ea) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.902284 2381 ts_tablet_manager.cc:1434] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20251026 02:56:21.903637 2378 tablet_bootstrap.cc:654] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.905367 2381 tablet_bootstrap.cc:492] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:21.905929 2376 tablet_replica.cc:333] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:21.906087 2376 raft_consensus.cc:2243] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.906256 2376 raft_consensus.cc:2272] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.907160 2381 tablet_bootstrap.cc:654] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.907121 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4a7c7fde2faf424590cdf3e2ddf0c1a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:21.907557 2376 ts_tablet_manager.cc:1916] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:21.907668 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a7c7fde2faf424590cdf3e2ddf0c1a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:21.912269 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3ef7247d62ec4adc82fdbed40fc852ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef7247d62ec4adc82fdbed40fc852ea already in progress: creating tablet
I20251026 02:56:21.913026 2377 ts_tablet_manager.cc:1916] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.913167 2375 tablet_bootstrap.cc:492] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:21.913357 2375 ts_tablet_manager.cc:1403] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.076s user 0.012s sys 0.014s
I20251026 02:56:21.914271 2375 raft_consensus.cc:359] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.914480 2375 raft_consensus.cc:385] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.914598 2375 raft_consensus.cc:740] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.914861 2375 consensus_queue.cc:260] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:21.915704 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad55040f0a2b4cfa894d1bd058d4d69f has no consensus state
I20251026 02:56:21.917551 2375 ts_tablet_manager.cc:1434] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:21.919114 1660 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet ac451c9166804a28893e0567f341a86d
I20251026 02:56:21.919445 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_03f29b21370a4ee8b75f3f37f6666e75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:21.919663 2375 tablet_bootstrap.cc:492] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:21.920022 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03f29b21370a4ee8b75f3f37f6666e75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.920998 2376 ts_tablet_manager.cc:1929] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.921547 2375 tablet_bootstrap.cc:654] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.923525 2376 log.cc:1199] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/64d00ae95e14439eaa748d17ca67118a
I20251026 02:56:21.924165 2376 ts_tablet_manager.cc:1950] T 64d00ae95e14439eaa748d17ca67118a P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:21.926019 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 64d00ae95e14439eaa748d17ca67118a (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
I20251026 02:56:21.927443 1654 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (table default.Table_537013d780884ee89ee2ad679f506a49 [id=095ad208529348d4aea2d4e2840e07c7]). Expected version 2 got 0
I20251026 02:56:21.928439 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 565255802f4f42ba8320a3ec80d2db8f
I20251026 02:56:21.929752 2378 tablet_bootstrap.cc:492] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:21.929876 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 565255802f4f42ba8320a3ec80d2db8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.929942 2378 ts_tablet_manager.cc:1403] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.009s
I20251026 02:56:21.930264 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 565255802f4f42ba8320a3ec80d2db8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.930779 2378 raft_consensus.cc:359] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.931015 2378 raft_consensus.cc:385] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.931686 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet cd8644f86972403b8477f97e69be6f80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.931893 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet b7154ef047d94a8ebc305bc3b08d108c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.932646 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 64d00ae95e14439eaa748d17ca67118a with delete_type TABLET_DATA_DELETED (replaced by tablet 3ef7247d62ec4adc82fdbed40fc852ea) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.933014 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 3802bec60e074107b9a8f8ddc3895fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.933892 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4ba66147c249728713a42d20f541f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 630beab92a0d4b928c79d9f0c32795f0) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.934199 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet ac43df5558634550a12aa65c521d8a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.934443 2372 tablet_replica.cc:333] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:21.934441 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet cd8644f86972403b8477f97e69be6f80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.934657 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 19edc0546fd04110b37353178da2652e with delete_type TABLET_DATA_DELETED (replaced by tablet ddfe9452fbe4426fb8f955b4bed586d9) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.934687 2372 raft_consensus.cc:2243] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.934914 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 565255802f4f42ba8320a3ec80d2db8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.934967 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 20c39a62bfab429bb231ae073585d0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.934988 2372 raft_consensus.cc:2272] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.935669 2372 ts_tablet_manager.cc:1916] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.936057 2376 tablet_replica.cc:333] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:21.936326 2376 raft_consensus.cc:2243] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.936498 2376 raft_consensus.cc:2272] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.936761 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7272209d4652989c3e6d8201a255 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.936965 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4ba66147c249728713a42d20f541f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 630beab92a0d4b928c79d9f0c32795f0) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.937150 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f4436f64997445aa880610c5a93f955 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.937304 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 3802bec60e074107b9a8f8ddc3895fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.937476 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet b7154ef047d94a8ebc305bc3b08d108c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.937666 2405 raft_consensus.cc:493] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.937796 2405 raft_consensus.cc:515] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.938328 2405 leader_election.cc:290] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.938769 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 64d00ae95e14439eaa748d17ca67118a with delete_type TABLET_DATA_DELETED (replaced by tablet 3ef7247d62ec4adc82fdbed40fc852ea) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.938890 2440 raft_consensus.cc:493] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.938998 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b68c0b860984df99a7eeb73854a5da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.939074 2440 raft_consensus.cc:515] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.939325 2376 ts_tablet_manager.cc:1916] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.939646 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet ac43df5558634550a12aa65c521d8a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:21.939743 2440 leader_election.cc:290] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.939942 2440 raft_consensus.cc:493] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.940075 2440 raft_consensus.cc:515] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.940376 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e56e5eec1474aefae04d278ded30765" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.940639 2440 leader_election.cc:290] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:21.941087 2194 leader_election.cc:343] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.941421 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e79223b1086347199470a2526e32bd6f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.941788 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd8644f86972403b8477f97e69be6f80" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:21.943816 2194 leader_election.cc:343] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:21.944048 2194 leader_election.cc:343] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: Tablet not RUNNING: STOPPED
W20251026 02:56:21.944554 1647 catalog_manager.cc:5101] could not convert tablet's 760e6c761a7b49f695308fa1da9eda37 schema from PB: OK
I20251026 02:56:21.945015 2440 raft_consensus.cc:493] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:21.945062 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 565255802f4f42ba8320a3ec80d2db8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 565255802f4f42ba8320a3ec80d2db8f already in progress: creating tablet
I20251026 02:56:21.945196 2440 raft_consensus.cc:515] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.945842 2440 leader_election.cc:290] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.946592 2376 ts_tablet_manager.cc:1929] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:21.946749 2376 log.cc:1199] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/cd8644f86972403b8477f97e69be6f80
I20251026 02:56:21.947124 2376 ts_tablet_manager.cc:1950] T cd8644f86972403b8477f97e69be6f80 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:21.948009 2376 tablet_replica.cc:333] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:21.948226 2376 raft_consensus.cc:2243] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:21.948417 2376 raft_consensus.cc:2272] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:21.948478 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a04ac20b7624a018c403ef4f64f7a69" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.948973 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9f25820b797a4156aa4bfd227c3a0087"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.949565 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f25820b797a4156aa4bfd227c3a0087 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.949651 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e56e5eec1474aefae04d278ded30765" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:21.949940 2194 leader_election.cc:343] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.950176 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd8644f86972403b8477f97e69be6f80" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:21.950141 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_765fd8cb95b14c77976811056434b451"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:21.950273 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet cd8644f86972403b8477f97e69be6f80 (table default.Table_d15145ea85264fc9aab94645c7ff3b41 [id=9d7e6bf7d8844942a0221a92da270f3c]) successfully deleted
I20251026 02:56:21.950567 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e79223b1086347199470a2526e32bd6f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:21.950745 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_765fd8cb95b14c77976811056434b451 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.950910 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f43d7cfe30cd40c08a88e4a62da5cbf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:21.951591 2195 leader_election.cc:343] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.951633 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a04ac20b7624a018c403ef4f64f7a69" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:21.951714 2195 leader_election.cc:304] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:21.952178 2195 leader_election.cc:343] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.952247 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6b07e3e332c141aa824ea90a76f40208" } modify_external_catalogs: true
I20251026 02:56:21.952343 2195 leader_election.cc:304] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.952409 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6b07e3e332c141aa824ea90a76f40208" } modify_external_catalogs: true
W20251026 02:56:21.952733 2195 leader_election.cc:343] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.952854 2195 leader_election.cc:304] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.953260 2440 raft_consensus.cc:2749] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:21.953455 2440 raft_consensus.cc:2749] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:21.953660 2440 raft_consensus.cc:2749] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:21.955374 2375 tablet_bootstrap.cc:492] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:21.955472 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d029698b06db4a6f807451acb73a3680"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:21.955811 2375 ts_tablet_manager.cc:1403] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.011s
I20251026 02:56:21.955845 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 20c39a62bfab429bb231ae073585d0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:21.956017 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d029698b06db4a6f807451acb73a3680 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.956749 2375 raft_consensus.cc:359] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.956955 2375 raft_consensus.cc:385] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:21.957046 2375 raft_consensus.cc:740] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.957247 2378 raft_consensus.cc:740] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:21.957336 2375 consensus_queue.cc:260] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:21.958484 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f43d7cfe30cd40c08a88e4a62da5cbf4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:21.960551 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7272209d4652989c3e6d8201a255 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.961627 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet b7154ef047d94a8ebc305bc3b08d108c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.961874 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet de24439a6a3a4734a08160cb7393429a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.963573 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet cd8644f86972403b8477f97e69be6f80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:21.964161 2375 ts_tablet_manager.cc:1434] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251026 02:56:21.957531 2378 consensus_queue.cc:260] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:21.964938 2195 leader_election.cc:343] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.965093 2195 leader_election.cc:304] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:21.965384 2407 raft_consensus.cc:2749] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:21.967067 2098 tablet_service.cc:1505] Processing CreateTablet for tablet b435520cadb344edaa29975b4e19718a (DEFAULT_TABLE table=default.Table_2eb5a7f69fbd436bb6e965caafc50691 [id=72fa811ef7224d42a83e602c3707f028]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.967584 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b435520cadb344edaa29975b4e19718a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.967772 1958 tablet_service.cc:1505] Processing CreateTablet for tablet b435520cadb344edaa29975b4e19718a (DEFAULT_TABLE table=default.Table_2eb5a7f69fbd436bb6e965caafc50691 [id=72fa811ef7224d42a83e602c3707f028]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.968267 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b435520cadb344edaa29975b4e19718a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.970880 2376 ts_tablet_manager.cc:1916] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:21.973752 2407 raft_consensus.cc:493] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:21.973948 2407 raft_consensus.cc:515] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.974745 2407 leader_election.cc:290] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:21.975746 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bc92f90a68b4e749bfd4fa49076bfe2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:21.976554 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_673a4fc67d4f4457b78f63d90414a2d3" } new_table_name: "default.Table_0ffb653ff26d42cdb0276a492f332424" modify_external_catalogs: true
W20251026 02:56:21.976929 2195 leader_election.cc:343] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:21.980368 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 2ee0a408253e4f2b942c510ae64c703e (DEFAULT_TABLE table=default.Table_c550a959e98a4e22853deca526de3eae [id=47bcd084e7fa4dec9f23cef12d10aa46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.980628 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 81530bbf4a2d4e908add683475e594fb (DEFAULT_TABLE table=default.Table_99b42c4a4a874b68bb4beba4c08cdeb0 [id=472d44e7e6f0409ab9d238c4913cc8a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.980835 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ee0a408253e4f2b942c510ae64c703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.981042 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81530bbf4a2d4e908add683475e594fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.982683 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 2ee0a408253e4f2b942c510ae64c703e (DEFAULT_TABLE table=default.Table_c550a959e98a4e22853deca526de3eae [id=47bcd084e7fa4dec9f23cef12d10aa46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.983170 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ee0a408253e4f2b942c510ae64c703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.983168 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 20c39a62bfab429bb231ae073585d0e5 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:21.984555 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 81530bbf4a2d4e908add683475e594fb (DEFAULT_TABLE table=default.Table_99b42c4a4a874b68bb4beba4c08cdeb0 [id=472d44e7e6f0409ab9d238c4913cc8a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.984907 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81530bbf4a2d4e908add683475e594fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.985706 2236 tablet_service.cc:1505] Processing CreateTablet for tablet b435520cadb344edaa29975b4e19718a (DEFAULT_TABLE table=default.Table_2eb5a7f69fbd436bb6e965caafc50691 [id=72fa811ef7224d42a83e602c3707f028]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.986085 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b435520cadb344edaa29975b4e19718a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.986732 2375 tablet_bootstrap.cc:492] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:21.987757 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 829a6044a2da44ff90acdebca6d5b18d (DEFAULT_TABLE table=default.Table_673a4fc67d4f4457b78f63d90414a2d3 [id=b8be7adba1cd48ae979de506d892cc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.988164 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829a6044a2da44ff90acdebca6d5b18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.988529 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 829a6044a2da44ff90acdebca6d5b18d (DEFAULT_TABLE table=default.Table_673a4fc67d4f4457b78f63d90414a2d3 [id=b8be7adba1cd48ae979de506d892cc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.988870 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829a6044a2da44ff90acdebca6d5b18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:21.992992 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 19edc0546fd04110b37353178da2652e with delete_type TABLET_DATA_DELETED (replaced by tablet ddfe9452fbe4426fb8f955b4bed586d9) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:21.993453 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 829a6044a2da44ff90acdebca6d5b18d (DEFAULT_TABLE table=default.Table_673a4fc67d4f4457b78f63d90414a2d3 [id=b8be7adba1cd48ae979de506d892cc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.994294 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 2ee0a408253e4f2b942c510ae64c703e (DEFAULT_TABLE table=default.Table_c550a959e98a4e22853deca526de3eae [id=47bcd084e7fa4dec9f23cef12d10aa46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.994594 2375 tablet_bootstrap.cc:654] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:21.995906 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 81530bbf4a2d4e908add683475e594fb (DEFAULT_TABLE table=default.Table_99b42c4a4a874b68bb4beba4c08cdeb0 [id=472d44e7e6f0409ab9d238c4913cc8a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:21.996584 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bc92f90a68b4e749bfd4fa49076bfe2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:21.996829 1982 raft_consensus.cc:2468] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:21.999930 2372 ts_tablet_manager.cc:1929] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.001526 2242 tablet_service.cc:1505] Processing CreateTablet for tablet ad55040f0a2b4cfa894d1bd058d4d69f (DEFAULT_TABLE table=default.Table_0da51595453549b3b7e077f7b7ff3ec7 [id=53cd517e68e64597ad51d85d16f0399d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.002094 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad55040f0a2b4cfa894d1bd058d4d69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.004469 2407 raft_consensus.cc:493] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.004712 2407 raft_consensus.cc:515] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:21.976554 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0da51595453549b3b7e077f7b7ff3ec7" } modify_external_catalogs: true
I20251026 02:56:22.005565 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0da51595453549b3b7e077f7b7ff3ec7" } modify_external_catalogs: true
I20251026 02:56:22.005577 2407 leader_election.cc:290] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:22.001183 2091 tablet_service.cc:1505] Processing CreateTablet for tablet ad55040f0a2b4cfa894d1bd058d4d69f (DEFAULT_TABLE table=default.Table_0da51595453549b3b7e077f7b7ff3ec7 [id=53cd517e68e64597ad51d85d16f0399d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.006649 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad55040f0a2b4cfa894d1bd058d4d69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.006951 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7154ef047d94a8ebc305bc3b08d108c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:22.007036 2194 leader_election.cc:304] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:22.007203 2117 raft_consensus.cc:2468] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:22.007864 2407 raft_consensus.cc:2804] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:22.007925 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7154ef047d94a8ebc305bc3b08d108c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:22.007987 2407 raft_consensus.cc:493] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:22.008108 2407 raft_consensus.cc:3060] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.010316 2375 tablet_bootstrap.cc:492] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:22.010391 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 565255802f4f42ba8320a3ec80d2db8f from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:22.010468 2375 ts_tablet_manager.cc:1403] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.008s
I20251026 02:56:22.010546 2372 log.cc:1199] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3a4ba66147c249728713a42d20f541f4
I20251026 02:56:22.011039 2372 ts_tablet_manager.cc:1950] T 3a4ba66147c249728713a42d20f541f4 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:22.011101 2194 leader_election.cc:343] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: Tablet not RUNNING: STOPPED
I20251026 02:56:22.011415 2375 raft_consensus.cc:359] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.011502 1962 tablet_service.cc:1505] Processing CreateTablet for tablet ad55040f0a2b4cfa894d1bd058d4d69f (DEFAULT_TABLE table=default.Table_0da51595453549b3b7e077f7b7ff3ec7 [id=53cd517e68e64597ad51d85d16f0399d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.011647 2375 raft_consensus.cc:385] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.011742 2375 raft_consensus.cc:740] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.012241 2195 leader_election.cc:304] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:22.012688 2405 raft_consensus.cc:2804] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:22.012843 2405 raft_consensus.cc:493] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:22.012981 2405 raft_consensus.cc:3060] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:22.014184 2195 consensus_peers.cc:597] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:22.016059 2405 raft_consensus.cc:515] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.017153 2405 leader_election.cc:290] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:22.018113 2375 consensus_queue.cc:260] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.020350 1981 raft_consensus.cc:1275] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:22.021019 2440 consensus_queue.cc:1048] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:22.021355 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7154ef047d94a8ebc305bc3b08d108c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:22.021549 2117 raft_consensus.cc:3060] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.021719 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3a4ba66147c249728713a42d20f541f4 (table default.Table_448f8c6a1d0c4125af909cf6570e41ae [id=9e895e2eb27749839df30c205507d222]) successfully deleted
I20251026 02:56:22.022109 2372 tablet_replica.cc:333] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:22.022087 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7154ef047d94a8ebc305bc3b08d108c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:22.022318 2372 raft_consensus.cc:2243] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:22.024657 2194 leader_election.cc:343] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: Tablet not RUNNING: STOPPED
I20251026 02:56:22.028009 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829a6044a2da44ff90acdebca6d5b18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.028703 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81530bbf4a2d4e908add683475e594fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.029158 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad55040f0a2b4cfa894d1bd058d4d69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.029605 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ee0a408253e4f2b942c510ae64c703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.029873 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 5e56e5eec1474aefae04d278ded30765
I20251026 02:56:22.030551 1657 catalog_manager.cc:6719] ReplaceTablet: tablet 5e56e5eec1474aefae04d278ded30765 deleted and replaced by tablet 23fd1284c9cb41bd98209ccbfe342bca
I20251026 02:56:22.030647 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 2ee0a408253e4f2b942c510ae64c703e
I20251026 02:56:22.032008 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 760e6c761a7b49f695308fa1da9eda37
I20251026 02:56:22.033342 2378 ts_tablet_manager.cc:1434] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.103s user 0.001s sys 0.001s
I20251026 02:56:22.033524 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 79942752cba4429abdc9524f882800da (table default.Table_a5ff6167974b47798ffe2b3cd583b50b [id=43cb7f538f824ae38059645230cc7e94]). Expected version 1 got 0
I20251026 02:56:22.033702 2378 tablet_bootstrap.cc:492] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:22.035619 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a72502f837fb4542b1f00bab5da37724"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.036417 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a72502f837fb4542b1f00bab5da37724 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:22.038600 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 829a6044a2da44ff90acdebca6d5b18d consensus state has no leader
I20251026 02:56:22.011109 2407 raft_consensus.cc:515] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.040732 2407 leader_election.cc:290] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:22.041332 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bc92f90a68b4e749bfd4fa49076bfe2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:22.041565 1982 raft_consensus.cc:3060] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.045097 1982 raft_consensus.cc:2468] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:22.045603 2194 leader_election.cc:304] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:22.046104 2407 raft_consensus.cc:2804] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:22.046514 2407 raft_consensus.cc:697] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:22.046912 2407 consensus_queue.cc:237] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.052649 2378 tablet_bootstrap.cc:654] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.053454 2377 ts_tablet_manager.cc:1929] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:22.057936 2376 ts_tablet_manager.cc:1929] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.062575 2372 raft_consensus.cc:2272] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.066128 2375 ts_tablet_manager.cc:1434] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20251026 02:56:22.066594 2375 tablet_bootstrap.cc:492] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:22.071367 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b4b762fde2224cb9b578e257daac19b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.072203 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4b762fde2224cb9b578e257daac19b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.073648 2376 log.cc:1199] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/b7154ef047d94a8ebc305bc3b08d108c
I20251026 02:56:22.074148 2376 ts_tablet_manager.cc:1950] T b7154ef047d94a8ebc305bc3b08d108c P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:22.063361 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet 760e6c761a7b49f695308fa1da9eda37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.075426 2376 tablet_replica.cc:333] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:22.075464 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_689f802c9e4b457eb71fb78898fe14b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:22.075627 2376 raft_consensus.cc:2243] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.075775 2376 raft_consensus.cc:2272] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:22.076006 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_689f802c9e4b457eb71fb78898fe14b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.077301 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet b7154ef047d94a8ebc305bc3b08d108c (table default.Table_c6bad2c857bb4c9fb8f8e2f5a87344cb [id=6fd49e190e4b4b2cba4b2fc5744b9f63]) successfully deleted
W20251026 02:56:22.077677 1647 catalog_manager.cc:4952] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 64d00ae95e14439eaa748d17ca67118a because the tablet was not found. No further retry: Not found: Tablet not found: 64d00ae95e14439eaa748d17ca67118a
I20251026 02:56:22.078294 2376 ts_tablet_manager.cc:1916] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.080392 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ad55040f0a2b4cfa894d1bd058d4d69f
I20251026 02:56:22.081881 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet ad55040f0a2b4cfa894d1bd058d4d69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.065948 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ee0a408253e4f2b942c510ae64c703e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.067713 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 3791cb18fe1f4937badb4859d00647f3 (DEFAULT_TABLE table=default.Table_6e720c10b85f4046a9617986f013c678 [id=c7a6636ca94a4863a89fbef09c339bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.085525 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3791cb18fe1f4937badb4859d00647f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.086422 2372 ts_tablet_manager.cc:1916] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.086741 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bc92f90a68b4e749bfd4fa49076bfe2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:22.087419 2376 ts_tablet_manager.cc:1929] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.087580 2376 log.cc:1199] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3802bec60e074107b9a8f8ddc3895fe2
I20251026 02:56:22.087953 2376 ts_tablet_manager.cc:1950] T 3802bec60e074107b9a8f8ddc3895fe2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:22.087906 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ee0a408253e4f2b942c510ae64c703e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.088433 2376 tablet_replica.cc:333] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:22.088620 2376 raft_consensus.cc:2243] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.088811 2376 raft_consensus.cc:2272] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:22.093458 1916 consensus_peers.cc:597] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:22.067692 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e56e5eec1474aefae04d278ded30765 with delete_type TABLET_DATA_DELETED (replaced by tablet 23fd1284c9cb41bd98209ccbfe342bca) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.072333 2378 tablet_bootstrap.cc:492] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:22.094341 2378 ts_tablet_manager.cc:1403] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.003s
I20251026 02:56:22.095253 2378 raft_consensus.cc:359] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.095476 2378 raft_consensus.cc:385] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.095574 2378 raft_consensus.cc:740] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.095816 2378 consensus_queue.cc:260] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.096387 2378 ts_tablet_manager.cc:1434] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:22.096722 2378 tablet_bootstrap.cc:492] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:22.099588 2378 tablet_bootstrap.cc:654] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.104440 2117 raft_consensus.cc:2468] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:22.106439 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet ad55040f0a2b4cfa894d1bd058d4d69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.106950 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 3791cb18fe1f4937badb4859d00647f3 (DEFAULT_TABLE table=default.Table_6e720c10b85f4046a9617986f013c678 [id=c7a6636ca94a4863a89fbef09c339bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.107367 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 760e6c761a7b49f695308fa1da9eda37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.107844 2375 tablet_bootstrap.cc:654] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.108248 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3791cb18fe1f4937badb4859d00647f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.060305 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e56e5eec1474aefae04d278ded30765 with delete_type TABLET_DATA_DELETED (replaced by tablet 23fd1284c9cb41bd98209ccbfe342bca) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.065562 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 760e6c761a7b49f695308fa1da9eda37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.065758 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ee0a408253e4f2b942c510ae64c703e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.069923 2262 raft_consensus.cc:1275] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Refusing update from remote peer 7fef01f4f2b843538ea245a09dd087a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:22.073288 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 3791cb18fe1f4937badb4859d00647f3 (DEFAULT_TABLE table=default.Table_6e720c10b85f4046a9617986f013c678 [id=c7a6636ca94a4863a89fbef09c339bc1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:22.087539 2195 leader_election.cc:343] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.110391 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3791cb18fe1f4937badb4859d00647f3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:22.111256 2195 consensus_peers.cc:597] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:22.111500 2195 leader_election.cc:304] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:22.112051 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet ad55040f0a2b4cfa894d1bd058d4d69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.112353 2407 raft_consensus.cc:2804] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:22.112510 2407 raft_consensus.cc:697] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:22.112905 2407 consensus_queue.cc:237] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.114300 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet e79223b1086347199470a2526e32bd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.114509 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 426ce6fb80874df68210a652f7b7602f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.115118 2452 consensus_queue.cc:1048] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 02:56:22.116456 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fe6cbe999c54b299d729e97f0f938e7 with delete_type TABLET_DATA_DELETED (replaced by tablet ad55040f0a2b4cfa894d1bd058d4d69f) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.116953 2377 log.cc:1199] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/de24439a6a3a4734a08160cb7393429a
I20251026 02:56:22.117749 2377 ts_tablet_manager.cc:1950] T de24439a6a3a4734a08160cb7393429a P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:22.122504 2452 raft_consensus.cc:493] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.122687 2452 raft_consensus.cc:515] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.123385 2452 leader_election.cc:290] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.126998 2376 ts_tablet_manager.cc:1916] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.127297 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e79223b1086347199470a2526e32bd6f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:22.129011 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3802bec60e074107b9a8f8ddc3895fe2 (table default.Table_1b8997fc8d6b4f788983c76e84b2d0c5 [id=37382ab63e7d448788deaa4cd95c8b44]) successfully deleted
W20251026 02:56:22.127838 1916 leader_election.cc:343] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.130386 1624 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
/tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.150.60:33729
--webserver_interface=127.1.150.60
--webserver_port=39671
--builtin_ntp_servers=127.1.150.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251026 02:56:22.124168 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e79223b1086347199470a2526e32bd6f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:22.130749 2263 raft_consensus.cc:2468] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:22.131390 1915 leader_election.cc:304] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:22.132001 2452 raft_consensus.cc:2804] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:22.132124 2452 raft_consensus.cc:493] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:22.132274 2452 raft_consensus.cc:3060] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.135432 2452 raft_consensus.cc:515] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.136701 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e79223b1086347199470a2526e32bd6f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:22.136787 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e79223b1086347199470a2526e32bd6f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:22.137023 2263 raft_consensus.cc:3060] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:22.137344 1916 leader_election.cc:343] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.140332 2452 leader_election.cc:290] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.140576 2263 raft_consensus.cc:2468] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:22.141144 1915 leader_election.cc:304] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:22.141741 2452 raft_consensus.cc:2804] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:22.141927 2452 raft_consensus.cc:697] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:22.142236 2452 consensus_queue.cc:237] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.144574 2372 ts_tablet_manager.cc:1929] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.145191 2372 log.cc:1199] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/ac43df5558634550a12aa65c521d8a14
I20251026 02:56:22.145840 2372 ts_tablet_manager.cc:1950] T ac43df5558634550a12aa65c521d8a14 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:22.147588 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet ac43df5558634550a12aa65c521d8a14 (table default.Table_8827f142daf64c45b7786c00d72ace16 [id=c881f3e0b92f45788e52269792793f47]) successfully deleted
I20251026 02:56:22.104287 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e56e5eec1474aefae04d278ded30765 with delete_type TABLET_DATA_DELETED (replaced by tablet 23fd1284c9cb41bd98209ccbfe342bca) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:22.149660 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 565255802f4f42ba8320a3ec80d2db8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 565255802f4f42ba8320a3ec80d2db8f already in progress: creating tablet
I20251026 02:56:22.150331 2372 tablet_replica.cc:333] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:22.151574 2372 raft_consensus.cc:2243] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.152392 2372 raft_consensus.cc:2272] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.154254 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2eb5a7f69fbd436bb6e965caafc50691" } new_table_name: "default.Table_60490d1326d642f1a1946ec2de119125" modify_external_catalogs: true
I20251026 02:56:22.156252 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cf81dba46e2b4f6da4bd597901aaddcb" } new_table_name: "default.Table_7e5c0b5970974e12aba66398ec59430e" modify_external_catalogs: true
I20251026 02:56:22.156293 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7badd06b09d543b39ca9a058c2ee80b5" } new_table_name: "default.Table_4890c6d7eb22457281042726080734db" modify_external_catalogs: true
I20251026 02:56:22.157159 2381 tablet_bootstrap.cc:492] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:22.157349 2381 ts_tablet_manager.cc:1403] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.252s user 0.025s sys 0.075s
I20251026 02:56:22.158334 2381 raft_consensus.cc:359] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.158412 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_17277a07df7c41a3a3b95c89e0459a5b" } new_table_name: "default.Table_58b97e74137344d982dd6cc88fb0348c" modify_external_catalogs: true
I20251026 02:56:22.158596 2381 raft_consensus.cc:385] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.158582 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f43d7cfe30cd40c08a88e4a62da5cbf4" } modify_external_catalogs: true
I20251026 02:56:22.158669 2381 raft_consensus.cc:740] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.158778 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f43d7cfe30cd40c08a88e4a62da5cbf4" } modify_external_catalogs: true
I20251026 02:56:22.159657 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_90929641047f40f1973c0e399816b5ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.160321 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90929641047f40f1973c0e399816b5ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.160339 2372 ts_tablet_manager.cc:1916] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.162374 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_71bb5e64488b4f4f92c57f6a1e34fb3c" } modify_external_catalogs: true
I20251026 02:56:22.162590 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_71bb5e64488b4f4f92c57f6a1e34fb3c" } modify_external_catalogs: true
I20251026 02:56:22.162758 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f8a501269b994d8c8864349d12974e16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:22.163262 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_095bb281b8804e38b5bd56a1dbabacfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.163390 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8a501269b994d8c8864349d12974e16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:22.163766 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_095bb281b8804e38b5bd56a1dbabacfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.165977 2377 tablet_replica.cc:333] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:22.166427 2377 raft_consensus.cc:2243] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.166684 2377 raft_consensus.cc:2272] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.167439 2377 ts_tablet_manager.cc:1916] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.172657 2381 consensus_queue.cc:260] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.174305 2378 tablet_bootstrap.cc:492] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:22.174621 2378 ts_tablet_manager.cc:1403] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.019s
W20251026 02:56:22.175173 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b435520cadb344edaa29975b4e19718a consensus state has no leader
I20251026 02:56:22.178321 2378 raft_consensus.cc:359] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.178766 2378 raft_consensus.cc:385] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.178921 2378 raft_consensus.cc:740] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.180065 2378 consensus_queue.cc:260] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.182111 2381 ts_tablet_manager.cc:1434] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20251026 02:56:22.183229 2378 ts_tablet_manager.cc:1434] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20251026 02:56:22.184464 2378 tablet_bootstrap.cc:492] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:22.184782 2372 ts_tablet_manager.cc:1929] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.184928 2372 log.cc:1199] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/604b7272209d4652989c3e6d8201a255
I20251026 02:56:22.185302 2372 ts_tablet_manager.cc:1950] T 604b7272209d4652989c3e6d8201a255 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:22.185626 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7036a159e88b4171942dd238f8af3ed3 has no consensus state
I20251026 02:56:22.186690 2381 tablet_bootstrap.cc:492] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:22.188604 2381 tablet_bootstrap.cc:654] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.191144 2375 tablet_bootstrap.cc:492] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:22.191344 2375 ts_tablet_manager.cc:1403] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.125s user 0.009s sys 0.005s
I20251026 02:56:22.192153 2375 raft_consensus.cc:359] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.192394 2375 raft_consensus.cc:385] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.192472 2375 raft_consensus.cc:740] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.192749 2375 consensus_queue.cc:260] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.193085 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 604b7272209d4652989c3e6d8201a255 (table default.Table_e00202a12c9349eda8d47f735e4cddab [id=d7ac91b3b98441baaa6ecc48cb904c7e]) successfully deleted
I20251026 02:56:22.195094 2375 ts_tablet_manager.cc:1434] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:22.196272 2375 tablet_bootstrap.cc:492] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:22.197680 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7439c051ca634fbe8b511d47ccede2ff has no consensus state
I20251026 02:56:22.197994 2375 tablet_bootstrap.cc:654] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.200126 2372 tablet_replica.cc:333] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:22.200418 2372 raft_consensus.cc:2243] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.200675 2372 raft_consensus.cc:2272] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.201478 2372 ts_tablet_manager.cc:1916] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.207829 2381 tablet_bootstrap.cc:492] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:22.208025 2381 ts_tablet_manager.cc:1403] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251026 02:56:22.179026 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet de24439a6a3a4734a08160cb7393429a (table default.Table_3c222bf9316b4ffe89d6899f4ded8349 [id=931430a4a03741bd86175eabf9764a77]) successfully deleted
W20251026 02:56:22.210650 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3ef7247d62ec4adc82fdbed40fc852ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef7247d62ec4adc82fdbed40fc852ea already in progress: creating tablet
W20251026 02:56:22.211125 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 565255802f4f42ba8320a3ec80d2db8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 565255802f4f42ba8320a3ec80d2db8f already in progress: creating tablet
I20251026 02:56:22.211876 2381 raft_consensus.cc:359] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:22.212097 2381 raft_consensus.cc:385] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.212244 2381 raft_consensus.cc:740] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.213496 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_17afc2f758be4959a0fceca2a296e3e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.214124 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17afc2f758be4959a0fceca2a296e3e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.215368 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8d12147552b34b2e95477a27d64104b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.215807 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d12147552b34b2e95477a27d64104b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.216426 2378 tablet_bootstrap.cc:654] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.224359 2381 consensus_queue.cc:260] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:22.225745 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b108f0d3dae84fab93bff012400c4a2a consensus state has no leader
I20251026 02:56:22.226205 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fbe94f0457844195a37ef907c6b74182"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:22.226059 2491 tablet.cc:1722] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251026 02:56:22.226750 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbe94f0457844195a37ef907c6b74182 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.228461 2375 tablet_bootstrap.cc:492] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:22.228628 2375 ts_tablet_manager.cc:1403] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20251026 02:56:22.229441 2375 raft_consensus.cc:359] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.229669 2375 raft_consensus.cc:385] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.229773 2375 raft_consensus.cc:740] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.230041 2375 consensus_queue.cc:260] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.231416 2452 raft_consensus.cc:493] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.231724 2452 raft_consensus.cc:515] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.232708 2375 ts_tablet_manager.cc:1434] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:22.233847 2375 tablet_bootstrap.cc:492] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:22.233958 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19edc0546fd04110b37353178da2652e" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:22.234165 2263 raft_consensus.cc:2468] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:22.234617 1915 leader_election.cc:304] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:22.235145 2393 raft_consensus.cc:2804] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:22.235244 2393 raft_consensus.cc:493] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:22.235343 2393 raft_consensus.cc:3060] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.235293 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d2dcec579c684b92a92d98fb59ee3da1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:22.235610 2375 tablet_bootstrap.cc:654] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:22.235877 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2dcec579c684b92a92d98fb59ee3da1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.236593 2387 raft_consensus.cc:493] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.236733 2387 raft_consensus.cc:515] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.237316 2387 leader_election.cc:290] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.237349 2381 ts_tablet_manager.cc:1434] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.029s user 0.002s sys 0.001s
I20251026 02:56:22.237931 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19edc0546fd04110b37353178da2652e" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:22.238217 2393 raft_consensus.cc:515] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.238829 2393 leader_election.cc:290] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.239344 2381 tablet_bootstrap.cc:492] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:22.239629 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19edc0546fd04110b37353178da2652e" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:22.239828 2263 raft_consensus.cc:3060] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:22.240103 1916 leader_election.cc:343] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.240818 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19edc0546fd04110b37353178da2652e" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:22.242965 2263 raft_consensus.cc:2468] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:22.244918 1915 leader_election.cc:304] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:22.245277 2393 raft_consensus.cc:2804] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 02:56:22.245241 2053 leader_election.cc:343] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: de24439a6a3a4734a08160cb7393429a
W20251026 02:56:22.245476 2053 leader_election.cc:343] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: de24439a6a3a4734a08160cb7393429a
I20251026 02:56:22.245622 2053 leader_election.cc:304] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:22.246127 2381 tablet_bootstrap.cc:654] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:22.248720 1916 leader_election.cc:343] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.249754 2387 raft_consensus.cc:2749] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:22.252518 2117 raft_consensus.cc:3060] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.248684 2452 leader_election.cc:290] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.260977 2372 ts_tablet_manager.cc:1929] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.261642 2372 log.cc:1199] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/b7154ef047d94a8ebc305bc3b08d108c
I20251026 02:56:22.262173 2372 ts_tablet_manager.cc:1950] T b7154ef047d94a8ebc305bc3b08d108c P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:22.266129 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 82315e4c5a554eeca5048a496498ca1f from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:22.269960 2497 tablet.cc:1722] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:22.270967 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 79942752cba4429abdc9524f882800da from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:22.285130 2490 tablet.cc:1722] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:22.287359 2377 ts_tablet_manager.cc:1929] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.286921 2378 tablet_bootstrap.cc:492] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:22.287508 2377 log.cc:1199] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/cd8644f86972403b8477f97e69be6f80
I20251026 02:56:22.287885 2377 ts_tablet_manager.cc:1950] T cd8644f86972403b8477f97e69be6f80 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:22.288965 2377 tablet_replica.cc:333] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:22.289117 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet cd8644f86972403b8477f97e69be6f80 (table default.Table_d15145ea85264fc9aab94645c7ff3b41 [id=9d7e6bf7d8844942a0221a92da270f3c]) successfully deleted
I20251026 02:56:22.289223 2377 raft_consensus.cc:2243] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.289422 2377 raft_consensus.cc:2272] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.290555 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7e5c0b5970974e12aba66398ec59430e" } modify_external_catalogs: true
I20251026 02:56:22.290796 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7e5c0b5970974e12aba66398ec59430e" } modify_external_catalogs: true
I20251026 02:56:22.291077 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet b7154ef047d94a8ebc305bc3b08d108c (table default.Table_c6bad2c857bb4c9fb8f8e2f5a87344cb [id=6fd49e190e4b4b2cba4b2fc5744b9f63]) successfully deleted
I20251026 02:56:22.291895 2372 tablet_replica.cc:333] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:22.292186 2372 raft_consensus.cc:2243] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.295593 1659 catalog_manager.cc:5649] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:22.296201 2407 raft_consensus.cc:493] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.296140 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (table default.Table_537013d780884ee89ee2ad679f506a49 [id=095ad208529348d4aea2d4e2840e07c7]). Expected version 2 got 0
I20251026 02:56:22.296447 2407 raft_consensus.cc:515] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.296671 2377 ts_tablet_manager.cc:1916] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.297142 2407 leader_election.cc:290] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:22.297803 2372 raft_consensus.cc:2272] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.300730 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 98fe44aab33b40eba819386c3bffca46 (DEFAULT_TABLE table=default.Table_9f25820b797a4156aa4bfd227c3a0087 [id=91d789aff2464f73b60e03b52126a396]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.301792 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 1c1596dd4394419a8bb6ea6a84be9fea (DEFAULT_TABLE table=default.Table_4a7c7fde2faf424590cdf3e2ddf0c1a5 [id=d9c0bfe3ef1145189ccef147212f787b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.302219 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79942752cba4429abdc9524f882800da" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:22.302584 2098 tablet_service.cc:1505] Processing CreateTablet for tablet fd1484434dd64f4bb2764268b3d3b218 (DEFAULT_TABLE table=default.Table_f43d7cfe30cd40c08a88e4a62da5cbf4 [id=d342cbe7a7cb44928ec73415b30a779e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.303993 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 7036a159e88b4171942dd238f8af3ed3 (DEFAULT_TABLE table=default.Table_7e5c0b5970974e12aba66398ec59430e [id=59edc20bb9be438c9ef00aafcd1c6376]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.304571 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 3bcd7dec9c2149069e48dcf5e564595e (DEFAULT_TABLE table=default.Table_797feb703f214aac9ea3dccac59e202a [id=5cbfb8198a8947b3abd1c0801e92f100]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.305074 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98fe44aab33b40eba819386c3bffca46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.308408 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bcd7dec9c2149069e48dcf5e564595e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.308694 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet 82315e4c5a554eeca5048a496498ca1f from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:22.312417 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c1596dd4394419a8bb6ea6a84be9fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.313149 2375 tablet_bootstrap.cc:492] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:22.313347 2375 ts_tablet_manager.cc:1403] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.009s
I20251026 02:56:22.314209 2375 raft_consensus.cc:359] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.314429 2375 raft_consensus.cc:385] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.314499 2375 raft_consensus.cc:740] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.314749 2378 ts_tablet_manager.cc:1403] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.002s
I20251026 02:56:22.314791 2375 consensus_queue.cc:260] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.315347 2375 ts_tablet_manager.cc:1434] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:22.315694 2375 tablet_bootstrap.cc:492] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:22.317410 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7036a159e88b4171942dd238f8af3ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.320384 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd1484434dd64f4bb2764268b3d3b218. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.322261 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet 9a04ac20b7624a018c403ef4f64f7a69 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:22.327248 2195 leader_election.cc:343] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.330852 2375 tablet_bootstrap.cc:654] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.340386 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d89d54d7dc874d7b8d7599cbe59b281a" } modify_external_catalogs: true
I20251026 02:56:22.340631 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d89d54d7dc874d7b8d7599cbe59b281a" } modify_external_catalogs: true
I20251026 02:56:22.346426 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 1d8cdf79a97149618bcaf0fefee1dfeb (DEFAULT_TABLE table=default.Table_5edbe48bb8ce41de84eaeaa4b4137fdb [id=690ec15818d34cdfa990c8fcd36a19cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.346899 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d8cdf79a97149618bcaf0fefee1dfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.350342 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 82315e4c5a554eeca5048a496498ca1f
I20251026 02:56:22.350826 1663 catalog_manager.cc:6719] ReplaceTablet: tablet 82315e4c5a554eeca5048a496498ca1f deleted and replaced by tablet 740f7c7f186f45ceaa4c40e76419e3a1
I20251026 02:56:22.351810 2372 ts_tablet_manager.cc:1916] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.352062 2407 raft_consensus.cc:493] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.352331 2440 raft_consensus.cc:493] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.354655 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 82315e4c5a554eeca5048a496498ca1f with delete_type TABLET_DATA_DELETED (replaced by tablet 740f7c7f186f45ceaa4c40e76419e3a1) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:22.356581 1647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20251026 02:56:22.357364 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b4b762fde2224cb9b578e257daac19b9" } new_table_name: "default.Table_e4fbc44faf984982986abfb227d6663d" modify_external_catalogs: true
I20251026 02:56:22.358678 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_698ae47452154028aa91996591984443"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.359206 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_698ae47452154028aa91996591984443 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.359928 2440 raft_consensus.cc:515] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:22.359817 2407 raft_consensus.cc:515] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.361624 2236 tablet_service.cc:1505] Processing CreateTablet for tablet fd1484434dd64f4bb2764268b3d3b218 (DEFAULT_TABLE table=default.Table_f43d7cfe30cd40c08a88e4a62da5cbf4 [id=d342cbe7a7cb44928ec73415b30a779e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.362457 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd1484434dd64f4bb2764268b3d3b218. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.363561 2407 leader_election.cc:290] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:22.363896 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 98fe44aab33b40eba819386c3bffca46 (DEFAULT_TABLE table=default.Table_9f25820b797a4156aa4bfd227c3a0087 [id=91d789aff2464f73b60e03b52126a396]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.364419 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98fe44aab33b40eba819386c3bffca46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.364640 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79942752cba4429abdc9524f882800da" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:22.366155 2440 leader_election.cc:290] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:22.366624 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 1c1596dd4394419a8bb6ea6a84be9fea (DEFAULT_TABLE table=default.Table_4a7c7fde2faf424590cdf3e2ddf0c1a5 [id=d9c0bfe3ef1145189ccef147212f787b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.367002 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c1596dd4394419a8bb6ea6a84be9fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.367262 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 4eea9b18e5094a22a51942f438a20eb4
I20251026 02:56:22.367013 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 82315e4c5a554eeca5048a496498ca1f with delete_type TABLET_DATA_DELETED (replaced by tablet 740f7c7f186f45ceaa4c40e76419e3a1) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.367628 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07280105a2b74ec68e27982414b12902" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:22.367987 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64d00ae95e14439eaa748d17ca67118a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:22.368978 2195 leader_election.cc:343] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:22.369834 2501 log.cc:927] Time spent T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20251026 02:56:22.369871 2387 raft_consensus.cc:493] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.370043 2387 raft_consensus.cc:515] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.370791 2387 leader_election.cc:290] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.370852 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_aa762aa5113a46a98da2f4ded4e23172" } modify_external_catalogs: true
I20251026 02:56:22.371034 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_aa762aa5113a46a98da2f4ded4e23172" } modify_external_catalogs: true
W20251026 02:56:22.371042 2195 leader_election.cc:343] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.371379 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eea9b18e5094a22a51942f438a20eb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.369738 2375 tablet_bootstrap.cc:492] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:22.372226 2375 ts_tablet_manager.cc:1403] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.002s
I20251026 02:56:22.375958 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 1c1596dd4394419a8bb6ea6a84be9fea (DEFAULT_TABLE table=default.Table_4a7c7fde2faf424590cdf3e2ddf0c1a5 [id=d9c0bfe3ef1145189ccef147212f787b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.376544 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20c39a62bfab429bb231ae073585d0e5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:22.378789 1649 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 5e56e5eec1474aefae04d278ded30765 (table default.Table_83bdf2cfef4042748553ebe8ae34996a [id=6734e5731cc94e71b43c76512f649d57], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5e56e5eec1474aefae04d278ded30765 consensus state has no leader
I20251026 02:56:22.379449 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 3bcd7dec9c2149069e48dcf5e564595e (DEFAULT_TABLE table=default.Table_797feb703f214aac9ea3dccac59e202a [id=5cbfb8198a8947b3abd1c0801e92f100]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.380399 2262 raft_consensus.cc:2468] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:22.375124 2496 mvcc.cc:204] Tried to move back new op lower bound from 7214888476580003840 to 7214888474270728192. Current Snapshot: MvccSnapshot[applied={T|T < 7214888476580003840}]
I20251026 02:56:22.382866 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 98fe44aab33b40eba819386c3bffca46 (DEFAULT_TABLE table=default.Table_9f25820b797a4156aa4bfd227c3a0087 [id=91d789aff2464f73b60e03b52126a396]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:22.383108 2194 leader_election.cc:343] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.383265 2194 leader_election.cc:304] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:22.383672 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 1d8cdf79a97149618bcaf0fefee1dfeb (DEFAULT_TABLE table=default.Table_5edbe48bb8ce41de84eaeaa4b4137fdb [id=690ec15818d34cdfa990c8fcd36a19cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.384028 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7036a159e88b4171942dd238f8af3ed3
I20251026 02:56:22.384061 2440 raft_consensus.cc:2749] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:22.385246 2393 raft_consensus.cc:697] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:22.385229 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 7036a159e88b4171942dd238f8af3ed3 (DEFAULT_TABLE table=default.Table_7e5c0b5970974e12aba66398ec59430e [id=59edc20bb9be438c9ef00aafcd1c6376]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.385202 2499 tablet.cc:1722] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:22.385543 2393 consensus_queue.cc:237] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.385850 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c1596dd4394419a8bb6ea6a84be9fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.385931 1961 tablet_service.cc:1505] Processing CreateTablet for tablet fd1484434dd64f4bb2764268b3d3b218 (DEFAULT_TABLE table=default.Table_f43d7cfe30cd40c08a88e4a62da5cbf4 [id=d342cbe7a7cb44928ec73415b30a779e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.386277 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd1484434dd64f4bb2764268b3d3b218. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.386577 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20c39a62bfab429bb231ae073585d0e5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:22.387665 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 7036a159e88b4171942dd238f8af3ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.387974 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07280105a2b74ec68e27982414b12902" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:22.388243 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c46d7af7b60a48188efbea88a1b7a72b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:22.388422 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 3bcd7dec9c2149069e48dcf5e564595e (DEFAULT_TABLE table=default.Table_797feb703f214aac9ea3dccac59e202a [id=5cbfb8198a8947b3abd1c0801e92f100]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.388780 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 82315e4c5a554eeca5048a496498ca1f with delete_type TABLET_DATA_DELETED (replaced by tablet 740f7c7f186f45ceaa4c40e76419e3a1) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:22.388916 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c46d7af7b60a48188efbea88a1b7a72b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.389289 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 7439c051ca634fbe8b511d47ccede2ff (DEFAULT_TABLE table=default.Table_4890c6d7eb22457281042726080734db [id=354a2ac9115e4f2385537f1c16992882]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.383831 2378 raft_consensus.cc:359] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.389972 2378 raft_consensus.cc:385] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.390127 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 7036a159e88b4171942dd238f8af3ed3 (DEFAULT_TABLE table=default.Table_7e5c0b5970974e12aba66398ec59430e [id=59edc20bb9be438c9ef00aafcd1c6376]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.386957 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 1d8cdf79a97149618bcaf0fefee1dfeb (DEFAULT_TABLE table=default.Table_5edbe48bb8ce41de84eaeaa4b4137fdb [id=690ec15818d34cdfa990c8fcd36a19cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.389787 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 7439c051ca634fbe8b511d47ccede2ff (DEFAULT_TABLE table=default.Table_4890c6d7eb22457281042726080734db [id=354a2ac9115e4f2385537f1c16992882]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.391044 2378 raft_consensus.cc:740] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.391280 2378 consensus_queue.cc:260] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.391764 2378 ts_tablet_manager.cc:1434] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.077s user 0.002s sys 0.000s
I20251026 02:56:22.392082 2378 tablet_bootstrap.cc:492] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:22.392313 2053 leader_election.cc:304] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:22.392901 2387 raft_consensus.cc:493] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.393486 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd (DEFAULT_TABLE table=default.Table_d029698b06db4a6f807451acb73a3680 [id=2ae19615aa104e2b8791b86755e1184f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.394227 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98fe44aab33b40eba819386c3bffca46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.396019 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7036a159e88b4171942dd238f8af3ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.398379 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eea9b18e5094a22a51942f438a20eb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.401700 2376 ts_tablet_manager.cc:1929] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.401830 2376 log.cc:1199] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/20c39a62bfab429bb231ae073585d0e5
I20251026 02:56:22.402050 2387 raft_consensus.cc:515] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.402222 2376 ts_tablet_manager.cc:1950] T 20c39a62bfab429bb231ae073585d0e5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:22.402246 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 79942752cba4429abdc9524f882800da
I20251026 02:56:22.402627 2376 tablet_replica.cc:333] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:22.402709 1658 catalog_manager.cc:6719] ReplaceTablet: tablet 79942752cba4429abdc9524f882800da deleted and replaced by tablet 9a08e8b7eb1949408f2151f2d7257e9d
I20251026 02:56:22.402772 2387 leader_election.cc:290] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.402936 2387 raft_consensus.cc:2804] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:22.403014 2387 raft_consensus.cc:493] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:22.403067 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7439c051ca634fbe8b511d47ccede2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.403139 2387 raft_consensus.cc:3060] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.398660 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 7036a159e88b4171942dd238f8af3ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.407135 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 20c39a62bfab429bb231ae073585d0e5 (table default.Table_0aa77747f2be46d8be9e7c5873940ca1 [id=a414f870eb244e0ba5d2569c6c91edc0]) successfully deleted
I20251026 02:56:22.407224 2387 raft_consensus.cc:515] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.407382 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d8cdf79a97149618bcaf0fefee1dfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.407931 2387 leader_election.cc:290] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.408092 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e70cf6d4e0dc4017b484753f4b0250d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.408672 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e70cf6d4e0dc4017b484753f4b0250d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.409153 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 79942752cba4429abdc9524f882800da with delete_type TABLET_DATA_DELETED (replaced by tablet 9a08e8b7eb1949408f2151f2d7257e9d) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.410099 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eea9b18e5094a22a51942f438a20eb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.410712 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 7036a159e88b4171942dd238f8af3ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.411293 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bcd7dec9c2149069e48dcf5e564595e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.412667 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d8cdf79a97149618bcaf0fefee1dfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.413668 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7439c051ca634fbe8b511d47ccede2ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:22.414053 2194 consensus_peers.cc:597] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac43df5558634550a12aa65c521d8a14. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:22.415175 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.415067 2505 tablet.cc:1722] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:22.416114 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 7439c051ca634fbe8b511d47ccede2ff (DEFAULT_TABLE table=default.Table_4890c6d7eb22457281042726080734db [id=354a2ac9115e4f2385537f1c16992882]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.416719 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd (DEFAULT_TABLE table=default.Table_d029698b06db4a6f807451acb73a3680 [id=2ae19615aa104e2b8791b86755e1184f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.416770 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet 79942752cba4429abdc9524f882800da with delete_type TABLET_DATA_DELETED (replaced by tablet 9a08e8b7eb1949408f2151f2d7257e9d) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.417632 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.420202 1654 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (table default.Table_537013d780884ee89ee2ad679f506a49 [id=095ad208529348d4aea2d4e2840e07c7]). Expected version 2 got 0
I20251026 02:56:22.420331 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd (DEFAULT_TABLE table=default.Table_d029698b06db4a6f807451acb73a3680 [id=2ae19615aa104e2b8791b86755e1184f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.420656 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7036a159e88b4171942dd238f8af3ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.421137 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bcd7dec9c2149069e48dcf5e564595e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.423897 2375 raft_consensus.cc:359] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.424106 2375 raft_consensus.cc:385] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.424033 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_60490d1326d642f1a1946ec2de119125" } new_table_name: "default.Table_66dd770512e847f4a14ee04e5afa3a6c" modify_external_catalogs: true
I20251026 02:56:22.425525 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4890c6d7eb22457281042726080734db" } new_table_name: "default.Table_be7d5ddd5e4e4b15862de09dc573f7f9" modify_external_catalogs: true
I20251026 02:56:22.426522 2381 tablet_bootstrap.cc:492] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:22.426601 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 79942752cba4429abdc9524f882800da with delete_type TABLET_DATA_DELETED (replaced by tablet 9a08e8b7eb1949408f2151f2d7257e9d) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:22.426829 2194 leader_election.cc:343] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.427170 2375 raft_consensus.cc:740] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.428403 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 0c647782049243d6bcd595f1a10d89c6 (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.429014 2381 ts_tablet_manager.cc:1403] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.187s user 0.007s sys 0.012s
I20251026 02:56:22.429759 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7439c051ca634fbe8b511d47ccede2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.429752 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 0c647782049243d6bcd595f1a10d89c6 (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.430382 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c647782049243d6bcd595f1a10d89c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.431026 2381 raft_consensus.cc:359] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.431998 2381 raft_consensus.cc:385] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.432101 2381 raft_consensus.cc:740] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.430099 2096 tablet_service.cc:1505] Processing CreateTablet for tablet ea910e98d17340efb816a8aa90f0ba7e (DEFAULT_TABLE table=default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.433243 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea910e98d17340efb816a8aa90f0ba7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.433611 2381 consensus_queue.cc:260] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.434162 2381 ts_tablet_manager.cc:1434] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251026 02:56:22.434998 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet fd1484434dd64f4bb2764268b3d3b218
I20251026 02:56:22.435196 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 0c647782049243d6bcd595f1a10d89c6 (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.436545 2194 leader_election.cc:304] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:22.436995 2194 leader_election.cc:343] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 64d00ae95e14439eaa748d17ca67118a
I20251026 02:56:22.437443 2194 leader_election.cc:304] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:22.438549 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c647782049243d6bcd595f1a10d89c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.438555 2405 raft_consensus.cc:2749] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:22.438796 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet fd1484434dd64f4bb2764268b3d3b218 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.432586 2375 consensus_queue.cc:260] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.439329 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet fd1484434dd64f4bb2764268b3d3b218 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.439635 2440 raft_consensus.cc:2749] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:22.440910 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_58b97e74137344d982dd6cc88fb0348c" } new_table_name: "default.Table_a2cb67c7ac66411f96b0ad63afcbd469" modify_external_catalogs: true
I20251026 02:56:22.442368 2375 ts_tablet_manager.cc:1434] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.066s user 0.000s sys 0.002s
I20251026 02:56:22.442656 2375 tablet_bootstrap.cc:492] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:22.445122 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet fd1484434dd64f4bb2764268b3d3b218 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.445951 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.447346 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 3d6690e6bf67454398637d9908b9c652 (DEFAULT_TABLE table=default.Table_e071f7a8a409464d810f12e647a4ff0e [id=c98c6fb4910449b0ad4513fc1c8f6f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.447742 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d6690e6bf67454398637d9908b9c652. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.447815 2233 tablet_service.cc:1505] Processing CreateTablet for tablet ea910e98d17340efb816a8aa90f0ba7e (DEFAULT_TABLE table=default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.449448 1956 tablet_service.cc:1505] Processing CreateTablet for tablet ea910e98d17340efb816a8aa90f0ba7e (DEFAULT_TABLE table=default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.453267 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9a04ac20b7624a018c403ef4f64f7a69
I20251026 02:56:22.453791 1656 catalog_manager.cc:6719] ReplaceTablet: tablet 9a04ac20b7624a018c403ef4f64f7a69 deleted and replaced by tablet 340bf8c78a88413299b61a52f0a578f3
I20251026 02:56:22.454124 2381 tablet_bootstrap.cc:492] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:22.456867 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea910e98d17340efb816a8aa90f0ba7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.457325 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a04ac20b7624a018c403ef4f64f7a69 with delete_type TABLET_DATA_DELETED (replaced by tablet 340bf8c78a88413299b61a52f0a578f3) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.457875 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c647782049243d6bcd595f1a10d89c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:22.458218 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 76306.3us
I20251026 02:56:22.459044 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a04ac20b7624a018c403ef4f64f7a69 with delete_type TABLET_DATA_DELETED (replaced by tablet 340bf8c78a88413299b61a52f0a578f3) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.459759 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9c97bccbf4cb4d15a66dbb8130acb6d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:22.460351 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20c39a62bfab429bb231ae073585d0e5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
W20251026 02:56:22.460422 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c97bccbf4cb4d15a66dbb8130acb6d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:22.462005 2053 leader_election.cc:343] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: Tablet not RUNNING: STOPPED
W20251026 02:56:22.463201 2053 leader_election.cc:343] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: cd8644f86972403b8477f97e69be6f80
I20251026 02:56:22.440711 2372 ts_tablet_manager.cc:1929] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.469360 2090 tablet_service.cc:1505] Processing CreateTablet for tablet d8ae08997af74b66b7d6375212eee4c2 (DEFAULT_TABLE table=default.Table_8c6fbc31d0674b8ebb6d80c95431b26f [id=789c86929bf94d2ca3fb5289f598e292]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.469911 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 20dec74440cf4bb9949c462ec51e9889 (DEFAULT_TABLE table=default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.469630 2238 tablet_service.cc:1505] Processing CreateTablet for tablet 20dec74440cf4bb9949c462ec51e9889 (DEFAULT_TABLE table=default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.470595 2262 raft_consensus.cc:3060] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.470793 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ae08997af74b66b7d6375212eee4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.471366 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7c32c2393a1144e9950b00f61ddba4eb
I20251026 02:56:22.474647 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c32c2393a1144e9950b00f61ddba4eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.478036 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20dec74440cf4bb9949c462ec51e9889. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.478221 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (table default.Table_537013d780884ee89ee2ad679f506a49 [id=095ad208529348d4aea2d4e2840e07c7]). Expected version 2 got 0
I20251026 02:56:22.480638 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd66abba7364951a7659011c71e76af with delete_type TABLET_DATA_DELETED (replaced by tablet 81530bbf4a2d4e908add683475e594fb) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.480930 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 20c39a62bfab429bb231ae073585d0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.480971 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 3d6690e6bf67454398637d9908b9c652 (DEFAULT_TABLE table=default.Table_e071f7a8a409464d810f12e647a4ff0e [id=c98c6fb4910449b0ad4513fc1c8f6f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.481702 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c32c2393a1144e9950b00f61ddba4eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.483772 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_50922eae951b497ba3b1e180c87facfa" } new_table_name: "default.Table_4578a4ce81ea42cd878859a8167babff" modify_external_catalogs: true
I20251026 02:56:22.484903 2375 tablet_bootstrap.cc:654] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.485684 2372 log.cc:1199] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/de24439a6a3a4734a08160cb7393429a
W20251026 02:56:22.490823 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cfed3523205548508d1101586ee3e2a9 has no consensus state
I20251026 02:56:22.492532 2372 ts_tablet_manager.cc:1950] T de24439a6a3a4734a08160cb7393429a P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:22.493611 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet de24439a6a3a4734a08160cb7393429a (table default.Table_3c222bf9316b4ffe89d6899f4ded8349 [id=931430a4a03741bd86175eabf9764a77]) successfully deleted
I20251026 02:56:22.494633 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_cdb10d198a394aac9033a98152fd2d94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.495290 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdb10d198a394aac9033a98152fd2d94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.496825 2372 tablet_replica.cc:333] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:22.497072 2372 raft_consensus.cc:2243] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.497435 2372 raft_consensus.cc:2272] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.498395 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef7247d62ec4adc82fdbed40fc852ea with delete_type TABLET_DATA_DELETED (replaced by tablet 0c647782049243d6bcd595f1a10d89c6) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.498625 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 07280105a2b74ec68e27982414b12902 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.499380 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20dec74440cf4bb9949c462ec51e9889. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.499521 2372 ts_tablet_manager.cc:1916] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.503921 2232 tablet_service.cc:1505] Processing CreateTablet for tablet d8ae08997af74b66b7d6375212eee4c2 (DEFAULT_TABLE table=default.Table_8c6fbc31d0674b8ebb6d80c95431b26f [id=789c86929bf94d2ca3fb5289f598e292]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.500056 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 8adf5c6970be46ed8434fd28404b2166 (DEFAULT_TABLE table=default.Table_03f29b21370a4ee8b75f3f37f6666e75 [id=e305b9beb6b74a33b3b7c64f34be8cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.504622 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8adf5c6970be46ed8434fd28404b2166. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.510380 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_25607539f64440cb8ff2c24a0a3a3bdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.510995 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25607539f64440cb8ff2c24a0a3a3bdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:22.511250 2053 leader_election.cc:343] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: cd8644f86972403b8477f97e69be6f80
I20251026 02:56:22.511420 2053 leader_election.cc:304] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:22.511605 2053 raft_consensus.cc:2686] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251026 02:56:22.511930 2053 leader_election.cc:343] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 20c39a62bfab429bb231ae073585d0e5
I20251026 02:56:22.513118 2375 tablet_bootstrap.cc:492] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:22.513317 2375 ts_tablet_manager.cc:1403] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.001s
I20251026 02:56:22.514101 2375 raft_consensus.cc:359] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.514318 2375 raft_consensus.cc:385] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.514390 2375 raft_consensus.cc:740] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.513871 2473 mvcc.cc:204] Tried to move back new op lower bound from 7214888476580003840 to 7214888474270728192. Current Snapshot: MvccSnapshot[applied={T|T < 7214888476580003840}]
I20251026 02:56:22.514629 1952 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a04ac20b7624a018c403ef4f64f7a69 with delete_type TABLET_DATA_DELETED (replaced by tablet 340bf8c78a88413299b61a52f0a578f3) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:22.514562 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 172952us
W20251026 02:56:22.518362 2194 consensus_peers.cc:597] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a4ba66147c249728713a42d20f541f4. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:22.520740 2376 raft_consensus.cc:2243] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:22.520823 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 8adf5c6970be46ed8434fd28404b2166 (DEFAULT_TABLE table=default.Table_03f29b21370a4ee8b75f3f37f6666e75 [id=e305b9beb6b74a33b3b7c64f34be8cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.520980 1949 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c32c2393a1144e9950b00f61ddba4eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.521317 2393 raft_consensus.cc:493] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.521515 2376 pending_rounds.cc:70] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: Trying to abort 1 pending ops.
I20251026 02:56:22.521440 2393 raft_consensus.cc:515] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.521672 2376 pending_rounds.cc:77] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888478301687808 op_type: NO_OP noop_request { }
I20251026 02:56:22.521792 2376 raft_consensus.cc:2889] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:22.521965 2376 raft_consensus.cc:2272] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.521999 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7b4f977cc4cc4d078f06c4251f8c4fa0
I20251026 02:56:22.521989 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 23fd1284c9cb41bd98209ccbfe342bca (DEFAULT_TABLE table=default.Table_83bdf2cfef4042748553ebe8ae34996a [id=6734e5731cc94e71b43c76512f649d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.520771 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 3d6690e6bf67454398637d9908b9c652 (DEFAULT_TABLE table=default.Table_e071f7a8a409464d810f12e647a4ff0e [id=c98c6fb4910449b0ad4513fc1c8f6f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.523270 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.523555 2375 consensus_queue.cc:260] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.520848 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 20dec74440cf4bb9949c462ec51e9889 (DEFAULT_TABLE table=default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.523967 2490 tablet.cc:1717] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539: Skipping requested alter to schema version 2, tablet already version 2
I20251026 02:56:22.526350 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e56e5eec1474aefae04d278ded30765" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:22.527456 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7814874203fc40af9ef07eeb2e11ebc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.528100 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7814874203fc40af9ef07eeb2e11ebc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.528816 2375 ts_tablet_manager.cc:1434] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251026 02:56:22.528802 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.529596 2375 tablet_bootstrap.cc:492] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:22.531816 2262 raft_consensus.cc:2468] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:22.532581 2053 leader_election.cc:304] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:22.523687 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 8adf5c6970be46ed8434fd28404b2166 (DEFAULT_TABLE table=default.Table_03f29b21370a4ee8b75f3f37f6666e75 [id=e305b9beb6b74a33b3b7c64f34be8cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.533008 2506 raft_consensus.cc:2804] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:22.533725 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23fd1284c9cb41bd98209ccbfe342bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.533702 2506 raft_consensus.cc:697] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:22.534139 2506 consensus_queue.cc:237] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.521212 1948 tablet_service.cc:1505] Processing CreateTablet for tablet d8ae08997af74b66b7d6375212eee4c2 (DEFAULT_TABLE table=default.Table_8c6fbc31d0674b8ebb6d80c95431b26f [id=789c86929bf94d2ca3fb5289f598e292]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.536001 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 23fd1284c9cb41bd98209ccbfe342bca (DEFAULT_TABLE table=default.Table_83bdf2cfef4042748553ebe8ae34996a [id=6734e5731cc94e71b43c76512f649d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.540802 2378 tablet_bootstrap.cc:654] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.541373 2375 tablet_bootstrap.cc:654] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:22.545176 1916 leader_election.cc:343] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.545531 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.545712 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea910e98d17340efb816a8aa90f0ba7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.546334 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 64d00ae95e14439eaa748d17ca67118a with delete_type TABLET_DATA_DELETED (replaced by tablet 3ef7247d62ec4adc82fdbed40fc852ea) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.546677 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_af080a4fb1da461eb6bf487aa8e54990"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.547394 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af080a4fb1da461eb6bf487aa8e54990 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.548449 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d6690e6bf67454398637d9908b9c652. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.549017 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet e79223b1086347199470a2526e32bd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.550094 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ae08997af74b66b7d6375212eee4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.550930 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 23fd1284c9cb41bd98209ccbfe342bca (DEFAULT_TABLE table=default.Table_83bdf2cfef4042748553ebe8ae34996a [id=6734e5731cc94e71b43c76512f649d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.550971 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e56e5eec1474aefae04d278ded30765" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:22.551366 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23fd1284c9cb41bd98209ccbfe342bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.551378 2262 raft_consensus.cc:2468] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:22.553274 2372 ts_tablet_manager.cc:1929] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.554366 2372 log.cc:1199] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/cd8644f86972403b8477f97e69be6f80
I20251026 02:56:22.555305 2372 ts_tablet_manager.cc:1950] T cd8644f86972403b8477f97e69be6f80 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:22.557950 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet cd8644f86972403b8477f97e69be6f80 (table default.Table_d15145ea85264fc9aab94645c7ff3b41 [id=9d7e6bf7d8844942a0221a92da270f3c]) successfully deleted
W20251026 02:56:22.560678 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 2ee0a408253e4f2b942c510ae64c703e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ee0a408253e4f2b942c510ae64c703e already in progress: creating tablet
W20251026 02:56:22.562309 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ad55040f0a2b4cfa894d1bd058d4d69f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad55040f0a2b4cfa894d1bd058d4d69f already in progress: creating tablet
I20251026 02:56:22.562680 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_65488c1f07e542b5973be5cfd122787c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:22.563345 2372 tablet_replica.cc:333] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:22.563340 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65488c1f07e542b5973be5cfd122787c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.563591 2372 raft_consensus.cc:2243] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.563781 2372 raft_consensus.cc:2272] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:22.565830 2507 log.cc:927] Time spent T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20251026 02:56:22.566416 2507 consensus_queue.cc:1175] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7fef01f4f2b843538ea245a09dd087a8" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20251026 02:56:22.567981 2376 ts_tablet_manager.cc:1916] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.570292 2393 leader_election.cc:290] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:22.573396 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b435520cadb344edaa29975b4e19718a consensus state has no leader
I20251026 02:56:22.575029 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8adf5c6970be46ed8434fd28404b2166. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.576421 2381 tablet_bootstrap.cc:654] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.577564 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8dcca91177f5452e816e2080bceadb93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:22.577826 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8adf5c6970be46ed8434fd28404b2166. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:22.578231 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dcca91177f5452e816e2080bceadb93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.580541 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23fd1284c9cb41bd98209ccbfe342bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.577294 1915 leader_election.cc:304] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:22.585976 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20dec74440cf4bb9949c462ec51e9889. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:22.586962 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7439c051ca634fbe8b511d47ccede2ff consensus state has no leader
I20251026 02:56:22.587215 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 2ee0a408253e4f2b942c510ae64c703e from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:22.588312 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d6690e6bf67454398637d9908b9c652. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.588465 2499 tablet.cc:1717] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8: Skipping requested alter to schema version 2, tablet already version 2
I20251026 02:56:22.591394 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4a7c7fde2faf424590cdf3e2ddf0c1a5" } modify_external_catalogs: true
I20251026 02:56:22.591573 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4a7c7fde2faf424590cdf3e2ddf0c1a5" } modify_external_catalogs: true
I20251026 02:56:22.591974 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ae08997af74b66b7d6375212eee4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.594754 2375 tablet_bootstrap.cc:492] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:22.594931 2375 ts_tablet_manager.cc:1403] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.012s
I20251026 02:56:22.595867 2375 raft_consensus.cc:359] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.596131 2375 raft_consensus.cc:385] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.596297 2375 raft_consensus.cc:740] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.599457 2375 consensus_queue.cc:260] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:22.602407 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b108f0d3dae84fab93bff012400c4a2a consensus state has no leader
I20251026 02:56:22.604290 2393 raft_consensus.cc:2804] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:22.604323 2375 ts_tablet_manager.cc:1434] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20251026 02:56:22.605389 2381 tablet_bootstrap.cc:492] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:22.605495 2393 raft_consensus.cc:493] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:22.605569 2381 ts_tablet_manager.cc:1403] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.153s user 0.003s sys 0.001s
I20251026 02:56:22.605629 2393 raft_consensus.cc:3060] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.605747 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_095bb281b8804e38b5bd56a1dbabacfa" } modify_external_catalogs: true
I20251026 02:56:22.605926 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_095bb281b8804e38b5bd56a1dbabacfa" } modify_external_catalogs: true
I20251026 02:56:22.606519 2381 raft_consensus.cc:359] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.606724 2381 raft_consensus.cc:385] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.606837 2381 raft_consensus.cc:740] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.607133 2381 consensus_queue.cc:260] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.607681 2381 ts_tablet_manager.cc:1434] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:22.608289 2381 tablet_bootstrap.cc:492] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:22.610268 2381 tablet_bootstrap.cc:654] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.612733 2505 tablet.cc:1717] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: Skipping requested alter to schema version 2, tablet already version 2
I20251026 02:56:22.623646 2377 ts_tablet_manager.cc:1929] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.623807 2377 log.cc:1199] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/19edc0546fd04110b37353178da2652e
W20251026 02:56:22.625687 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0f83d631faa468f828e54e97987a00b consensus state has no leader
I20251026 02:56:22.626844 2375 tablet_bootstrap.cc:492] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:22.632951 2377 ts_tablet_manager.cc:1950] T 19edc0546fd04110b37353178da2652e P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:22.635650 2377 tablet_replica.cc:333] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:22.636184 2377 raft_consensus.cc:2243] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.636776 2375 tablet_bootstrap.cc:654] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.637094 2377 raft_consensus.cc:2272] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.637965 2377 ts_tablet_manager.cc:1916] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.639046 2372 ts_tablet_manager.cc:1916] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.640151 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 19edc0546fd04110b37353178da2652e with delete_type TABLET_DATA_DELETED (replaced by tablet ddfe9452fbe4426fb8f955b4bed586d9) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.648654 2372 ts_tablet_manager.cc:1929] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:22.648912 2372 log.cc:1199] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/760e6c761a7b49f695308fa1da9eda37
I20251026 02:56:22.649431 2372 ts_tablet_manager.cc:1950] T 760e6c761a7b49f695308fa1da9eda37 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:22.650645 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 760e6c761a7b49f695308fa1da9eda37 (table default.Table_6b07e3e332c141aa824ea90a76f40208 [id=c3e9c585b810403f88e4eb2b4924b73c]) successfully deleted
I20251026 02:56:22.651371 2375 tablet_bootstrap.cc:492] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
W20251026 02:56:22.651414 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 5e56e5eec1474aefae04d278ded30765 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e56e5eec1474aefae04d278ded30765 already in progress: creating tablet
I20251026 02:56:22.651553 2375 ts_tablet_manager.cc:1403] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.001s
I20251026 02:56:22.651885 2372 tablet_replica.cc:333] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:22.652127 2372 raft_consensus.cc:2243] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.652428 2372 raft_consensus.cc:2272] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.653159 2375 raft_consensus.cc:359] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.653375 2375 raft_consensus.cc:385] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.653453 2375 raft_consensus.cc:740] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.653874 2372 ts_tablet_manager.cc:1916] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.660470 2375 consensus_queue.cc:260] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.664402 2393 raft_consensus.cc:515] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.666199 2375 ts_tablet_manager.cc:1434] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251026 02:56:22.669909 2375 tablet_bootstrap.cc:492] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:22.669463 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 3802bec60e074107b9a8f8ddc3895fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.673296 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e56e5eec1474aefae04d278ded30765" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:22.675331 2375 tablet_bootstrap.cc:654] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:22.676338 1916 leader_election.cc:343] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.678299 2377 ts_tablet_manager.cc:1929] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.678540 2377 log.cc:1199] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/20c39a62bfab429bb231ae073585d0e5
I20251026 02:56:22.678983 2377 ts_tablet_manager.cc:1950] T 20c39a62bfab429bb231ae073585d0e5 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:22.680416 2377 tablet_replica.cc:333] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:22.680683 2377 raft_consensus.cc:2243] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.685508 2393 leader_election.cc:290] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.685253 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e56e5eec1474aefae04d278ded30765" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:22.685798 2262 raft_consensus.cc:3060] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.688663 2262 raft_consensus.cc:2468] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:22.689704 1915 leader_election.cc:304] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:22.690292 2393 raft_consensus.cc:2804] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:22.691888 2393 raft_consensus.cc:697] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:22.692313 2393 consensus_queue.cc:237] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.693256 2381 tablet_bootstrap.cc:492] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:22.693436 2381 ts_tablet_manager.cc:1403] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.007s
I20251026 02:56:22.693817 2377 raft_consensus.cc:2272] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.694304 2381 raft_consensus.cc:359] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:22.694520 2381 raft_consensus.cc:385] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.694630 2381 raft_consensus.cc:740] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.694919 2381 consensus_queue.cc:260] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:22.695188 2377 ts_tablet_manager.cc:1916] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.697592 2376 ts_tablet_manager.cc:1929] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:22.697976 2376 log.cc:1199] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/19edc0546fd04110b37353178da2652e
I20251026 02:56:22.698196 2381 ts_tablet_manager.cc:1434] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251026 02:56:22.698469 2376 ts_tablet_manager.cc:1950] T 19edc0546fd04110b37353178da2652e P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:22.698516 2381 tablet_bootstrap.cc:492] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:22.699045 2372 ts_tablet_manager.cc:1929] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.700438 2381 tablet_bootstrap.cc:654] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.711373 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ff86fc06e7d46e78fe68cc6f31a30cc with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.715286 2377 ts_tablet_manager.cc:1929] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.715550 2377 log.cc:1199] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/604b7272209d4652989c3e6d8201a255
I20251026 02:56:22.716048 2377 ts_tablet_manager.cc:1950] T 604b7272209d4652989c3e6d8201a255 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:22.717209 2377 tablet_replica.cc:333] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:22.720312 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet b108f0d3dae84fab93bff012400c4a2a (table default.Table_a2cb67c7ac66411f96b0ad63afcbd469 [id=2f5d9104d21c49e0985a01ffb4b7a978]). Expected version 2 got 0
I20251026 02:56:22.720418 2377 raft_consensus.cc:2243] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:22.720743 2377 pending_rounds.cc:70] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:22.720942 2377 pending_rounds.cc:77] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888475480961024 op_type: NO_OP noop_request { }
I20251026 02:56:22.721143 2377 raft_consensus.cc:2889] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:22.721357 2377 raft_consensus.cc:2272] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.722009 2377 ts_tablet_manager.cc:1916] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.727258 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet b108f0d3dae84fab93bff012400c4a2a (table default.Table_a2cb67c7ac66411f96b0ad63afcbd469 [id=2f5d9104d21c49e0985a01ffb4b7a978]). Expected version 2 got 0
I20251026 02:56:22.729900 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1c1596dd4394419a8bb6ea6a84be9fea
I20251026 02:56:22.732972 2506 raft_consensus.cc:493] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.733516 2506 raft_consensus.cc:515] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.733641 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_bcf0aac1b41747e9a94a97b8430fa081" } modify_external_catalogs: true
I20251026 02:56:22.733842 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_bcf0aac1b41747e9a94a97b8430fa081" } modify_external_catalogs: true
I20251026 02:56:22.734247 2506 leader_election.cc:290] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.734479 2377 ts_tablet_manager.cc:1929] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:22.734614 2377 log.cc:1199] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3a4ba66147c249728713a42d20f541f4
I20251026 02:56:22.735091 2377 ts_tablet_manager.cc:1950] T 3a4ba66147c249728713a42d20f541f4 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:22.735203 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet 5e56e5eec1474aefae04d278ded30765 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:22.736011 2377 tablet_replica.cc:333] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:22.736065 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 9ec035b1caac450d9ab55e74a061ae38 (DEFAULT_TABLE table=default.Table_fbe94f0457844195a37ef907c6b74182 [id=e905acf634364d5896dc4cfbbbb27f77]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:22.736254 2053 leader_election.cc:343] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 19edc0546fd04110b37353178da2652e
I20251026 02:56:22.736496 2506 raft_consensus.cc:493] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.736619 2506 raft_consensus.cc:515] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.737011 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 8667ffa25f2b478a987826d486400bbe (DEFAULT_TABLE table=default.Table_17afc2f758be4959a0fceca2a296e3e1 [id=f7f1e852a46f4129b2709fd58a6be034]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.737187 2506 leader_election.cc:290] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.737491 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c1596dd4394419a8bb6ea6a84be9fea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.736629 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ec035b1caac450d9ab55e74a061ae38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.738461 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bc92f90a68b4e749bfd4fa49076bfe2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:22.738737 1979 raft_consensus.cc:2393] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b6b0a6c4e144982b67980118cddba48 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
W20251026 02:56:22.739454 2053 leader_election.cc:343] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 19edc0546fd04110b37353178da2652e
I20251026 02:56:22.739586 2053 leader_election.cc:304] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:22.739980 2506 raft_consensus.cc:2749] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:22.737785 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bc92f90a68b4e749bfd4fa49076bfe2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:22.740446 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 7176cbe8f64546b7871d047c0e0e5b2a (DEFAULT_TABLE table=default.Table_c46d7af7b60a48188efbea88a1b7a72b [id=93bef2a8d7b649e880042bfdbb1e08ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.740701 2053 leader_election.cc:304] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:22.741019 2372 log.cc:1199] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/82315e4c5a554eeca5048a496498ca1f
I20251026 02:56:22.741215 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet 0e89c69e3b58458aad10beaeb5128d60 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:22.741142 2506 raft_consensus.cc:3060] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.740559 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8667ffa25f2b478a987826d486400bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.741396 2377 raft_consensus.cc:2243] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.741582 2377 raft_consensus.cc:2272] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.742153 2377 ts_tablet_manager.cc:1916] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.743502 2376 tablet_replica.cc:333] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:22.743726 2376 raft_consensus.cc:2243] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.743922 2376 raft_consensus.cc:2272] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.744437 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d029698b06db4a6f807451acb73a3680" } modify_external_catalogs: true
I20251026 02:56:22.744658 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7176cbe8f64546b7871d047c0e0e5b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.744637 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d029698b06db4a6f807451acb73a3680" } modify_external_catalogs: true
I20251026 02:56:22.744748 2405 raft_consensus.cc:493] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.744933 2405 raft_consensus.cc:515] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:22.745604 2405 leader_election.cc:290] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:22.746234 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 0e89c69e3b58458aad10beaeb5128d60 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:22.746271 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b108f0d3dae84fab93bff012400c4a2a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:22.746495 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d9f69f33702146e281d6d9bfa43e968b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:22.746635 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a2cb67c7ac66411f96b0ad63afcbd469" } new_table_name: "default.Table_fa379fbb05ce4e46a08b7fc1bae3183a" modify_external_catalogs: true
W20251026 02:56:22.747251 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9f69f33702146e281d6d9bfa43e968b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.747442 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f8a501269b994d8c8864349d12974e16" } modify_external_catalogs: true
I20251026 02:56:22.747586 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f8a501269b994d8c8864349d12974e16" } modify_external_catalogs: true
I20251026 02:56:22.748404 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet c0f83d631faa468f828e54e97987a00b from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:22.748867 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ac4b46f3e31e4929bc07664386d206f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.749444 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac4b46f3e31e4929bc07664386d206f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.749490 2376 ts_tablet_manager.cc:1916] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:22.750742 2195 leader_election.cc:343] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.746649 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 19edc0546fd04110b37353178da2652e (table default.Table_f26f32ab73e848fea8718bfddbebb31f [id=aa6a3819787c4ab881cb8d71460a9b09]) successfully deleted
I20251026 02:56:22.751515 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 20c39a62bfab429bb231ae073585d0e5 (table default.Table_0aa77747f2be46d8be9e7c5873940ca1 [id=a414f870eb244e0ba5d2569c6c91edc0]) successfully deleted
I20251026 02:56:22.751917 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b108f0d3dae84fab93bff012400c4a2a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:22.752032 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 604b7272209d4652989c3e6d8201a255 (table default.Table_e00202a12c9349eda8d47f735e4cddab [id=d7ac91b3b98441baaa6ecc48cb904c7e]) successfully deleted
I20251026 02:56:22.752233 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3a4ba66147c249728713a42d20f541f4 (table default.Table_448f8c6a1d0c4125af909cf6570e41ae [id=9e895e2eb27749839df30c205507d222]) successfully deleted
I20251026 02:56:22.752419 2506 raft_consensus.cc:2749] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:22.752909 2194 leader_election.cc:343] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.753118 2194 leader_election.cc:304] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:22.753511 2405 raft_consensus.cc:2749] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:22.753968 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 19edc0546fd04110b37353178da2652e (table default.Table_f26f32ab73e848fea8718bfddbebb31f [id=aa6a3819787c4ab881cb8d71460a9b09]) successfully deleted
I20251026 02:56:22.755081 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 7176cbe8f64546b7871d047c0e0e5b2a (DEFAULT_TABLE table=default.Table_c46d7af7b60a48188efbea88a1b7a72b [id=93bef2a8d7b649e880042bfdbb1e08ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.755587 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7176cbe8f64546b7871d047c0e0e5b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.760298 2377 ts_tablet_manager.cc:1929] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.760550 2377 log.cc:1199] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3f4436f64997445aa880610c5a93f955
I20251026 02:56:22.760775 2378 tablet_bootstrap.cc:492] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:22.761008 2378 ts_tablet_manager.cc:1403] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.369s user 0.002s sys 0.009s
I20251026 02:56:22.761046 2377 ts_tablet_manager.cc:1950] T 3f4436f64997445aa880610c5a93f955 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:22.761860 2378 raft_consensus.cc:359] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.762076 2378 raft_consensus.cc:385] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.762147 2378 raft_consensus.cc:740] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.762457 2378 consensus_queue.cc:260] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.763265 2372 ts_tablet_manager.cc:1950] T 82315e4c5a554eeca5048a496498ca1f P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:22.763352 2378 ts_tablet_manager.cc:1434] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251026 02:56:22.763752 2378 tablet_bootstrap.cc:492] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:22.766844 2378 tablet_bootstrap.cc:654] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.766883 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 461412f3ad1e4186b1302cdf5a624ed4 (DEFAULT_TABLE table=default.Table_095bb281b8804e38b5bd56a1dbabacfa [id=6cab490620ef48f18ab808cd5d6c6af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.767380 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 461412f3ad1e4186b1302cdf5a624ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.768379 2377 tablet_replica.cc:333] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:22.768862 2377 raft_consensus.cc:2243] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.769119 2377 raft_consensus.cc:2272] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.769821 2377 ts_tablet_manager.cc:1916] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.770846 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 82315e4c5a554eeca5048a496498ca1f (table default.Table_cfeac8daaf08492a9456d2e06cd62a58 [id=1d09f5b9fa164ac783f0be9332eac4c3]) successfully deleted
I20251026 02:56:22.776278 2377 ts_tablet_manager.cc:1929] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251026 02:56:22.782287 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 4eea9b18e5094a22a51942f438a20eb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4eea9b18e5094a22a51942f438a20eb4 already in progress: creating tablet
W20251026 02:56:22.783113 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 79942752cba4429abdc9524f882800da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79942752cba4429abdc9524f882800da already in progress: creating tablet
I20251026 02:56:22.783555 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 461412f3ad1e4186b1302cdf5a624ed4
I20251026 02:56:22.780385 2405 raft_consensus.cc:493] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.783809 2405 raft_consensus.cc:515] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.784528 2405 leader_election.cc:290] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:22.784982 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ff86fc06e7d46e78fe68cc6f31a30cc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:22.785009 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 461412f3ad1e4186b1302cdf5a624ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.785332 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ff86fc06e7d46e78fe68cc6f31a30cc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:22.785508 2194 leader_election.cc:343] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:22.785959 2195 leader_election.cc:343] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.786078 2195 leader_election.cc:304] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:22.786209 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 0020cb1f190c49978c5322d387a47f71 (DEFAULT_TABLE table=default.Table_689f802c9e4b457eb71fb78898fe14b0 [id=05242557de9a4a19948b8591ce73ab28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.786586 2405 raft_consensus.cc:2749] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:22.786643 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0020cb1f190c49978c5322d387a47f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.786881 2377 log.cc:1199] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3802bec60e074107b9a8f8ddc3895fe2
I20251026 02:56:22.787326 2377 ts_tablet_manager.cc:1950] T 3802bec60e074107b9a8f8ddc3895fe2 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:22.788468 2377 tablet_replica.cc:333] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:22.788735 2377 raft_consensus.cc:2243] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:22.788866 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 7176cbe8f64546b7871d047c0e0e5b2a (DEFAULT_TABLE table=default.Table_c46d7af7b60a48188efbea88a1b7a72b [id=93bef2a8d7b649e880042bfdbb1e08ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.789021 2377 pending_rounds.cc:70] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:22.789104 2377 pending_rounds.cc:77] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888477137719296 op_type: NO_OP noop_request { }
I20251026 02:56:22.789222 2377 raft_consensus.cc:2889] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:22.789284 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7176cbe8f64546b7871d047c0e0e5b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.789347 2377 raft_consensus.cc:2272] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.791127 2377 ts_tablet_manager.cc:1916] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:22.792836 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7036a159e88b4171942dd238f8af3ed3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7036a159e88b4171942dd238f8af3ed3 already in progress: creating tablet
I20251026 02:56:22.800261 2377 ts_tablet_manager.cc:1929] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:22.800376 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet e0101098df05462bb26b3c187b259c08
I20251026 02:56:22.800537 2377 log.cc:1199] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/b7154ef047d94a8ebc305bc3b08d108c
I20251026 02:56:22.800513 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 9ec035b1caac450d9ab55e74a061ae38 (DEFAULT_TABLE table=default.Table_fbe94f0457844195a37ef907c6b74182 [id=e905acf634364d5896dc4cfbbbb27f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.801004 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c1596dd4394419a8bb6ea6a84be9fea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.801096 2377 ts_tablet_manager.cc:1950] T b7154ef047d94a8ebc305bc3b08d108c P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:22.801321 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c1596dd4394419a8bb6ea6a84be9fea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.801450 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ec035b1caac450d9ab55e74a061ae38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.802610 2377 tablet_replica.cc:333] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:22.802893 2377 raft_consensus.cc:2243] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.803476 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet e0101098df05462bb26b3c187b259c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.803944 1981 raft_consensus.cc:1275] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:22.804082 2117 raft_consensus.cc:1275] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:22.804663 2405 consensus_queue.cc:1048] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:22.804863 2440 consensus_queue.cc:1048] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:22.805823 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 8667ffa25f2b478a987826d486400bbe (DEFAULT_TABLE table=default.Table_17afc2f758be4959a0fceca2a296e3e1 [id=f7f1e852a46f4129b2709fd58a6be034]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.806263 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8667ffa25f2b478a987826d486400bbe. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:22.806651 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet fd1484434dd64f4bb2764268b3d3b218 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1484434dd64f4bb2764268b3d3b218 already in progress: creating tablet
I20251026 02:56:22.809752 2405 raft_consensus.cc:493] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.809906 2405 raft_consensus.cc:515] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.810529 2405 leader_election.cc:290] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:22.811271 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fe6cbe999c54b299d729e97f0f938e7" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:22.811374 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fe6cbe999c54b299d729e97f0f938e7" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:22.811836 2194 leader_election.cc:343] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:22.811993 2195 leader_election.cc:343] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.812119 2195 leader_election.cc:304] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:22.810779 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 9ec035b1caac450d9ab55e74a061ae38 (DEFAULT_TABLE table=default.Table_fbe94f0457844195a37ef907c6b74182 [id=e905acf634364d5896dc4cfbbbb27f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.812613 2405 raft_consensus.cc:2749] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:22.812669 2100 tablet_service.cc:1505] Processing CreateTablet for tablet cfed3523205548508d1101586ee3e2a9 (DEFAULT_TABLE table=default.Table_e4fbc44faf984982986abfb227d6663d [id=b35a8189bbc04e94bb097c9e13a38ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.812769 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ec035b1caac450d9ab55e74a061ae38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.813133 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfed3523205548508d1101586ee3e2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.817205 2377 raft_consensus.cc:2272] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.818084 2377 ts_tablet_manager.cc:1916] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.819376 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 8667ffa25f2b478a987826d486400bbe (DEFAULT_TABLE table=default.Table_17afc2f758be4959a0fceca2a296e3e1 [id=f7f1e852a46f4129b2709fd58a6be034]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.819765 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8667ffa25f2b478a987826d486400bbe. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:22.821866 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9a04ac20b7624a018c403ef4f64f7a69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a04ac20b7624a018c403ef4f64f7a69 already in progress: creating tablet
I20251026 02:56:22.822264 2376 ts_tablet_manager.cc:1929] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:22.827309 2375 tablet_bootstrap.cc:492] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:22.827473 2375 ts_tablet_manager.cc:1403] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.158s user 0.005s sys 0.024s
W20251026 02:56:22.828557 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7c32c2393a1144e9950b00f61ddba4eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c32c2393a1144e9950b00f61ddba4eb already in progress: creating tablet
I20251026 02:56:22.832094 2490 tablet.cc:1722] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:22.835011 1654 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (table default.Table_537013d780884ee89ee2ad679f506a49 [id=095ad208529348d4aea2d4e2840e07c7]). Expected version 2 got 0
I20251026 02:56:22.835469 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 461412f3ad1e4186b1302cdf5a624ed4 (DEFAULT_TABLE table=default.Table_095bb281b8804e38b5bd56a1dbabacfa [id=6cab490620ef48f18ab808cd5d6c6af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.835848 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 461412f3ad1e4186b1302cdf5a624ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.839505 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0e89c69e3b58458aad10beaeb5128d60
I20251026 02:56:22.839973 1659 catalog_manager.cc:6719] ReplaceTablet: tablet 0e89c69e3b58458aad10beaeb5128d60 deleted and replaced by tablet a821ed8a686c452a975acb96b5c74778
I20251026 02:56:22.841691 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (table default.Table_537013d780884ee89ee2ad679f506a49 [id=095ad208529348d4aea2d4e2840e07c7]). Expected version 2 got 0
I20251026 02:56:22.843418 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet 461412f3ad1e4186b1302cdf5a624ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.833724 2476 mvcc.cc:204] Tried to move back new op lower bound from 7214888479959949312 to 7214888476867305472. Current Snapshot: MvccSnapshot[applied={T|T < 7214888479959949312}]
I20251026 02:56:22.844409 2377 ts_tablet_manager.cc:1929] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.844470 2499 tablet.cc:1722] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:22.844552 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet e0101098df05462bb26b3c187b259c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.845046 2377 log.cc:1199] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/64d00ae95e14439eaa748d17ca67118a
I20251026 02:56:22.845151 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet e0101098df05462bb26b3c187b259c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.845443 2377 ts_tablet_manager.cc:1950] T 64d00ae95e14439eaa748d17ca67118a P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:22.846078 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3f4436f64997445aa880610c5a93f955 (table default.Table_7c60ae6cebc04434975a2c3782f4aaa7 [id=9d77f9be68084772a1f3ee4d00a5180d]) successfully deleted
I20251026 02:56:22.846284 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3802bec60e074107b9a8f8ddc3895fe2 (table default.Table_1b8997fc8d6b4f788983c76e84b2d0c5 [id=37382ab63e7d448788deaa4cd95c8b44]) successfully deleted
I20251026 02:56:22.846463 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3f60bcc5e4cb4fd19d2a5713f69814cd
I20251026 02:56:22.846585 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet b7154ef047d94a8ebc305bc3b08d108c (table default.Table_c6bad2c857bb4c9fb8f8e2f5a87344cb [id=6fd49e190e4b4b2cba4b2fc5744b9f63]) successfully deleted
I20251026 02:56:22.846943 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 64d00ae95e14439eaa748d17ca67118a (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
I20251026 02:56:22.840778 2378 tablet_bootstrap.cc:492] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:22.847828 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet 20dec74440cf4bb9949c462ec51e9889 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:22.847935 2378 ts_tablet_manager.cc:1403] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.002s
I20251026 02:56:22.848104 2377 tablet_replica.cc:333] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:22.848433 2393 raft_consensus.cc:493] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:22.848610 2393 raft_consensus.cc:515] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.848693 2377 raft_consensus.cc:2243] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.848891 2377 raft_consensus.cc:2272] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.848821 2378 raft_consensus.cc:359] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.848999 2378 raft_consensus.cc:385] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.849102 2378 raft_consensus.cc:740] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.850018 2378 consensus_queue.cc:260] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.850666 2378 ts_tablet_manager.cc:1434] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251026 02:56:22.850732 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 0020cb1f190c49978c5322d387a47f71 (DEFAULT_TABLE table=default.Table_689f802c9e4b457eb71fb78898fe14b0 [id=05242557de9a4a19948b8591ce73ab28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.850997 2378 tablet_bootstrap.cc:492] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:22.852083 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0020cb1f190c49978c5322d387a47f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.857580 2378 tablet_bootstrap.cc:654] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.843875 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 461412f3ad1e4186b1302cdf5a624ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.849310 2393 leader_election.cc:290] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.858842 2376 log.cc:1199] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/760e6c761a7b49f695308fa1da9eda37
I20251026 02:56:22.859380 2376 ts_tablet_manager.cc:1950] T 760e6c761a7b49f695308fa1da9eda37 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:22.841233 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 461412f3ad1e4186b1302cdf5a624ed4 (DEFAULT_TABLE table=default.Table_095bb281b8804e38b5bd56a1dbabacfa [id=6cab490620ef48f18ab808cd5d6c6af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.860248 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 461412f3ad1e4186b1302cdf5a624ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.865758 2376 tablet_replica.cc:333] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:22.865963 2376 raft_consensus.cc:2243] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:22.866242 2376 pending_rounds.cc:70] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8: Trying to abort 1 pending ops.
I20251026 02:56:22.866305 2376 pending_rounds.cc:77] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888479511629824 op_type: NO_OP noop_request { }
I20251026 02:56:22.866452 2376 raft_consensus.cc:2889] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:22.866567 2376 raft_consensus.cc:2272] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.871493 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 760e6c761a7b49f695308fa1da9eda37 (table default.Table_6b07e3e332c141aa824ea90a76f40208 [id=c3e9c585b810403f88e4eb2b4924b73c]) successfully deleted
W20251026 02:56:22.871742 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ad55040f0a2b4cfa894d1bd058d4d69f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad55040f0a2b4cfa894d1bd058d4d69f already in progress: creating tablet
W20251026 02:56:22.871963 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 2ee0a408253e4f2b942c510ae64c703e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ee0a408253e4f2b942c510ae64c703e already in progress: creating tablet
I20251026 02:56:22.872855 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.872993 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e89c69e3b58458aad10beaeb5128d60 with delete_type TABLET_DATA_DELETED (replaced by tablet a821ed8a686c452a975acb96b5c74778) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.873106 1948 tablet_service.cc:1505] Processing CreateTablet for tablet cfed3523205548508d1101586ee3e2a9 (DEFAULT_TABLE table=default.Table_e4fbc44faf984982986abfb227d6663d [id=b35a8189bbc04e94bb097c9e13a38ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.873550 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfed3523205548508d1101586ee3e2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.879805 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet c0f83d631faa468f828e54e97987a00b
I20251026 02:56:22.880501 1663 catalog_manager.cc:6719] ReplaceTablet: tablet c0f83d631faa468f828e54e97987a00b deleted and replaced by tablet 9156314f98ac4ed9981f6f13fc585043
I20251026 02:56:22.881088 2376 ts_tablet_manager.cc:1916] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.881322 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 78a624d839934d3089b64e397d4360af (DEFAULT_TABLE table=default.Table_a72502f837fb4542b1f00bab5da37724 [id=4db9b61c88b5422fade5d42038bc5bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.881857 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a624d839934d3089b64e397d4360af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.849416 2377 ts_tablet_manager.cc:1916] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.849721 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c32c2393a1144e9950b00f61ddba4eb" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:22.850208 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c32c2393a1144e9950b00f61ddba4eb" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:22.887634 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 78a624d839934d3089b64e397d4360af (DEFAULT_TABLE table=default.Table_a72502f837fb4542b1f00bab5da37724 [id=4db9b61c88b5422fade5d42038bc5bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.887745 2262 raft_consensus.cc:2468] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:22.888056 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a624d839934d3089b64e397d4360af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.859680 2381 tablet_bootstrap.cc:492] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:22.888309 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ac19c5a847454a23ab16831198ff2a20
I20251026 02:56:22.888463 2381 ts_tablet_manager.cc:1403] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.190s user 0.009s sys 0.017s
I20251026 02:56:22.889317 2381 raft_consensus.cc:359] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:22.889514 2381 raft_consensus.cc:385] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.889775 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet ac19c5a847454a23ab16831198ff2a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.889604 2381 raft_consensus.cc:740] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.890136 2381 consensus_queue.cc:260] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:22.890839 2381 ts_tablet_manager.cc:1434] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:22.891120 2381 tablet_bootstrap.cc:492] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:22.864739 2238 tablet_service.cc:1505] Processing CreateTablet for tablet 0020cb1f190c49978c5322d387a47f71 (DEFAULT_TABLE table=default.Table_689f802c9e4b457eb71fb78898fe14b0 [id=05242557de9a4a19948b8591ce73ab28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.891908 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0020cb1f190c49978c5322d387a47f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.865372 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e89c69e3b58458aad10beaeb5128d60 with delete_type TABLET_DATA_DELETED (replaced by tablet a821ed8a686c452a975acb96b5c74778) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.873298 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.893363 2099 tablet_service.cc:1505] Processing CreateTablet for tablet ac19c5a847454a23ab16831198ff2a20 (DEFAULT_TABLE table=default.Table_f8a501269b994d8c8864349d12974e16 [id=a48fa20d35e44980b60317a9c93b42d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.893771 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac19c5a847454a23ab16831198ff2a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.895596 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 7dec83bfb9e946db90c10d526af2dfca (DEFAULT_TABLE table=default.Table_90929641047f40f1973c0e399816b5ef [id=327c67aeb10146e08119f7f3a7a11769]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.896327 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dec83bfb9e946db90c10d526af2dfca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.900415 2381 tablet_bootstrap.cc:654] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.900936 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (DEFAULT_TABLE table=default.Table_cfeac8daaf08492a9456d2e06cd62a58 [id=1d09f5b9fa164ac783f0be9332eac4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.901345 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 740f7c7f186f45ceaa4c40e76419e3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.902915 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee (DEFAULT_TABLE table=default.Table_d2dcec579c684b92a92d98fb59ee3da1 [id=4f7edfb6d75c4a0b94ca99bb9cfe14f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.903399 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.853848 2375 raft_consensus.cc:359] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.903730 2375 raft_consensus.cc:385] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.903841 2375 raft_consensus.cc:740] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.904109 2375 consensus_queue.cc:260] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.905056 2375 ts_tablet_manager.cc:1434] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.077s user 0.003s sys 0.000s
I20251026 02:56:22.905366 2375 tablet_bootstrap.cc:492] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:22.865042 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.866500 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e89c69e3b58458aad10beaeb5128d60 with delete_type TABLET_DATA_DELETED (replaced by tablet a821ed8a686c452a975acb96b5c74778) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.867599 2372 tablet_replica.cc:333] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:22.870332 2513 tablet.cc:1722] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:22.873499 2242 tablet_service.cc:1505] Processing CreateTablet for tablet cfed3523205548508d1101586ee3e2a9 (DEFAULT_TABLE table=default.Table_e4fbc44faf984982986abfb227d6663d [id=b35a8189bbc04e94bb097c9e13a38ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.908360 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfed3523205548508d1101586ee3e2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.873299 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 78a624d839934d3089b64e397d4360af (DEFAULT_TABLE table=default.Table_a72502f837fb4542b1f00bab5da37724 [id=4db9b61c88b5422fade5d42038bc5bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.910220 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a624d839934d3089b64e397d4360af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.875370 2090 tablet_service.cc:1505] Processing CreateTablet for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 (DEFAULT_TABLE table=default.Table_698ae47452154028aa91996591984443 [id=9e897d33e6374ad4bc3c8acaae512645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.912508 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.884409 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet c0f83d631faa468f828e54e97987a00b with delete_type TABLET_DATA_DELETED (replaced by tablet 9156314f98ac4ed9981f6f13fc585043) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:22.917987 2233 tablet_service.cc:1505] Processing CreateTablet for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 (DEFAULT_TABLE table=default.Table_698ae47452154028aa91996591984443 [id=9e897d33e6374ad4bc3c8acaae512645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.918417 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.920461 2372 raft_consensus.cc:2243] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:22.920699 2372 raft_consensus.cc:2272] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.921409 2372 ts_tablet_manager.cc:1916] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.907503 2375 tablet_bootstrap.cc:654] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.925503 2239 tablet_service.cc:1505] Processing CreateTablet for tablet ac19c5a847454a23ab16831198ff2a20 (DEFAULT_TABLE table=default.Table_f8a501269b994d8c8864349d12974e16 [id=a48fa20d35e44980b60317a9c93b42d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.926267 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac19c5a847454a23ab16831198ff2a20. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:22.927439 1916 leader_election.cc:343] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.927862 2378 tablet_bootstrap.cc:492] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:22.928179 2378 ts_tablet_manager.cc:1403] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.002s
I20251026 02:56:22.928583 1915 leader_election.cc:304] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:22.928969 2381 tablet_bootstrap.cc:492] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:22.930169 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (DEFAULT_TABLE table=default.Table_cfeac8daaf08492a9456d2e06cd62a58 [id=1d09f5b9fa164ac783f0be9332eac4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.930663 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 7dec83bfb9e946db90c10d526af2dfca (DEFAULT_TABLE table=default.Table_90929641047f40f1973c0e399816b5ef [id=327c67aeb10146e08119f7f3a7a11769]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.930732 2393 raft_consensus.cc:2804] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:22.930827 2393 raft_consensus.cc:493] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:22.930981 2393 raft_consensus.cc:3060] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.933880 2393 raft_consensus.cc:515] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.934535 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet ac19c5a847454a23ab16831198ff2a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.935521 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c32c2393a1144e9950b00f61ddba4eb" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:22.936424 1951 tablet_service.cc:1505] Processing CreateTablet for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 (DEFAULT_TABLE table=default.Table_698ae47452154028aa91996591984443 [id=9e897d33e6374ad4bc3c8acaae512645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.936844 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.938977 2378 raft_consensus.cc:359] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.940057 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet ac19c5a847454a23ab16831198ff2a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:22.940261 1916 leader_election.cc:343] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:22.940019 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet c0f83d631faa468f828e54e97987a00b with delete_type TABLET_DATA_DELETED (replaced by tablet 9156314f98ac4ed9981f6f13fc585043) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:22.942278 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 7dec83bfb9e946db90c10d526af2dfca (DEFAULT_TABLE table=default.Table_90929641047f40f1973c0e399816b5ef [id=327c67aeb10146e08119f7f3a7a11769]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.942694 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dec83bfb9e946db90c10d526af2dfca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.944530 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (DEFAULT_TABLE table=default.Table_cfeac8daaf08492a9456d2e06cd62a58 [id=1d09f5b9fa164ac783f0be9332eac4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.944912 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 740f7c7f186f45ceaa4c40e76419e3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.946738 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee (DEFAULT_TABLE table=default.Table_d2dcec579c684b92a92d98fb59ee3da1 [id=4f7edfb6d75c4a0b94ca99bb9cfe14f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.947078 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.948480 1954 tablet_service.cc:1505] Processing CreateTablet for tablet ac19c5a847454a23ab16831198ff2a20 (DEFAULT_TABLE table=default.Table_f8a501269b994d8c8864349d12974e16 [id=a48fa20d35e44980b60317a9c93b42d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.948813 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac19c5a847454a23ab16831198ff2a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.948138 2377 ts_tablet_manager.cc:1929] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:22.952497 2378 raft_consensus.cc:385] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.952605 2378 raft_consensus.cc:740] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.952849 2378 consensus_queue.cc:260] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:22.953521 2378 ts_tablet_manager.cc:1434] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20251026 02:56:22.953828 2378 tablet_bootstrap.cc:492] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:22.954934 2377 log.cc:1199] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/0b68c0b860984df99a7eeb73854a5da8
I20251026 02:56:22.955528 2378 tablet_bootstrap.cc:654] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.955586 2377 ts_tablet_manager.cc:1950] T 0b68c0b860984df99a7eeb73854a5da8 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:22.956161 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 155a23a93358487a9564bb5ac175b4eb (DEFAULT_TABLE table=default.Table_8d12147552b34b2e95477a27d64104b6 [id=8203d96f9a7a4a6b9a69234b583600c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.956976 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 76) took 1021 ms. Trace:
I20251026 02:56:22.957361 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 155a23a93358487a9564bb5ac175b4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.960644 2393 leader_election.cc:290] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:22.961259 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet c0f83d631faa468f828e54e97987a00b with delete_type TABLET_DATA_DELETED (replaced by tablet 9156314f98ac4ed9981f6f13fc585043) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:22.961977 2375 tablet_bootstrap.cc:492] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:22.962023 2372 ts_tablet_manager.cc:1929] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251026 02:56:22.967105 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b108f0d3dae84fab93bff012400c4a2a consensus state has no leader
I20251026 02:56:22.968029 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_54ffd2743e604aa4bb1a694ca3217891" } new_table_name: "default.Table_e2223a0634ec4d9c96a847247e20a762" modify_external_catalogs: true
I20251026 02:56:22.957226 2377 rpcz_store.cc:276] 1026 02:56:21.935918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:21.938927 (+ 3009us) service_pool.cc:225] Handling call
1026 02:56:22.848020 (+909093us) ts_tablet_manager.cc:1151] Deleting tablet 0b68c0b860984df99a7eeb73854a5da8
1026 02:56:22.848028 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:22.921321 (+ 73293us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:22.946674 (+ 25353us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:22.956896 (+ 10222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":908935}
I20251026 02:56:22.969776 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8d12147552b34b2e95477a27d64104b6" } new_table_name: "default.Table_ac2590e433d341b48b9303f46840d3e5" modify_external_catalogs: true
I20251026 02:56:22.970099 2377 tablet_replica.cc:333] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:22.972666 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (table default.Table_537013d780884ee89ee2ad679f506a49 [id=095ad208529348d4aea2d4e2840e07c7]). Expected version 2 got 0
I20251026 02:56:22.973102 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7296727c5b2f4c95ac3ff62818bd84f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.973637 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7296727c5b2f4c95ac3ff62818bd84f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.974468 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ce7dea3ace314a5fa3ffeb9f16512725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.974948 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce7dea3ace314a5fa3ffeb9f16512725 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:22.975507 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 740f7c7f186f45ceaa4c40e76419e3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.975780 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dec83bfb9e946db90c10d526af2dfca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.976055 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ad061592630140d9952cf22072721a4d" } new_table_name: "default.Table_00dc4c49a19246889353df2157eebf15" modify_external_catalogs: true
I20251026 02:56:22.977288 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8c6fbc31d0674b8ebb6d80c95431b26f" } new_table_name: "default.Table_c095574955cf44d7aecfbca77de4ad1f" modify_external_catalogs: true
I20251026 02:56:22.980297 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c32c2393a1144e9950b00f61ddba4eb" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:22.980526 2262 raft_consensus.cc:3060] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:22.983454 2262 raft_consensus.cc:2468] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:22.984130 2381 ts_tablet_manager.cc:1403] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20251026 02:56:22.985013 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 9a08e8b7eb1949408f2151f2d7257e9d (DEFAULT_TABLE table=default.Table_a5ff6167974b47798ffe2b3cd583b50b [id=43cb7f538f824ae38059645230cc7e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.985199 2381 raft_consensus.cc:359] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:22.985407 2381 raft_consensus.cc:385] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:22.985376 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 0b68c0b860984df99a7eeb73854a5da8 (table default.Table_0d997fa5825e4cf2a797881f912b609f [id=319c8d881ba442c6a0629a979e01b289]) successfully deleted
I20251026 02:56:22.985482 2381 raft_consensus.cc:740] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:22.985633 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 155a23a93358487a9564bb5ac175b4eb (DEFAULT_TABLE table=default.Table_8d12147552b34b2e95477a27d64104b6 [id=8203d96f9a7a4a6b9a69234b583600c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.985718 2381 consensus_queue.cc:260] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:22.985996 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a08e8b7eb1949408f2151f2d7257e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.986271 2381 ts_tablet_manager.cc:1434] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:22.986567 2381 tablet_bootstrap.cc:492] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:22.988835 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee (DEFAULT_TABLE table=default.Table_d2dcec579c684b92a92d98fb59ee3da1 [id=4f7edfb6d75c4a0b94ca99bb9cfe14f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:22.990300 2381 tablet_bootstrap.cc:654] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:22.991706 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.992393 2377 raft_consensus.cc:2243] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:22.993337 2377 pending_rounds.cc:70] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:22.993541 2377 pending_rounds.cc:77] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888473850781696 op_type: NO_OP noop_request { }
I20251026 02:56:22.995224 2377 raft_consensus.cc:2889] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:22.995503 2377 raft_consensus.cc:2272] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:22.996320 2377 ts_tablet_manager.cc:1916] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.996658 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 155a23a93358487a9564bb5ac175b4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.998461 1663 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_537013d780884ee89ee2ad679f506a49 [id=095ad208529348d4aea2d4e2840e07c7] alter complete (version 2)
W20251026 02:56:22.998589 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2bfbe624075463f8b90ed4ec66b784c consensus state has no leader
I20251026 02:56:22.998970 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5fdecebb943046c599c1f24317ce85f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:22.999202 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 155a23a93358487a9564bb5ac175b4eb consensus state has no leader
I20251026 02:56:22.998992 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9709925e5dd846f78b29265c3a7635dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:22.999547 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_acad6a9e93e24617aff27d46e9c5c099"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.000072 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acad6a9e93e24617aff27d46e9c5c099 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:23.000078 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9709925e5dd846f78b29265c3a7635dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:22.999572 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fdecebb943046c599c1f24317ce85f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.001832 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5d136440d3b64581b112fb1a112950ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.002282 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d136440d3b64581b112fb1a112950ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.003777 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_953a107579574f9088605b5e07dd917e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:23.003860 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 9a08e8b7eb1949408f2151f2d7257e9d (DEFAULT_TABLE table=default.Table_a5ff6167974b47798ffe2b3cd583b50b [id=43cb7f538f824ae38059645230cc7e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.005460 1915 leader_election.cc:304] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:23.006057 2393 raft_consensus.cc:2804] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:23.009898 2393 raft_consensus.cc:697] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
W20251026 02:56:23.011327 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_953a107579574f9088605b5e07dd917e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.011766 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a08e8b7eb1949408f2151f2d7257e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.012815 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 340bf8c78a88413299b61a52f0a578f3 (DEFAULT_TABLE table=default.Table_627cb33ef7ce4082b858fa0670d09862 [id=c162ef89d2f14e91a24a6d8c2c37c261]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:23.014726 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a821ed8a686c452a975acb96b5c74778 has no consensus state
W20251026 02:56:23.014768 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8ae08997af74b66b7d6375212eee4c2 consensus state has no leader
I20251026 02:56:23.010242 2393 consensus_queue.cc:237] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.015506 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 20dec74440cf4bb9949c462ec51e9889
I20251026 02:56:23.015964 1660 catalog_manager.cc:6719] ReplaceTablet: tablet 20dec74440cf4bb9949c462ec51e9889 deleted and replaced by tablet c570e5f9a93c45bda32ec92c68794714
I20251026 02:56:23.018501 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340bf8c78a88413299b61a52f0a578f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.018878 2376 ts_tablet_manager.cc:1929] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:23.019220 2377 ts_tablet_manager.cc:1929] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:23.019374 2377 log.cc:1199] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/ac43df5558634550a12aa65c521d8a14
I20251026 02:56:23.019860 2377 ts_tablet_manager.cc:1950] T ac43df5558634550a12aa65c521d8a14 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.020736 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 74) took 1085 ms. Trace:
I20251026 02:56:23.020803 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 155a23a93358487a9564bb5ac175b4eb (DEFAULT_TABLE table=default.Table_8d12147552b34b2e95477a27d64104b6 [id=8203d96f9a7a4a6b9a69234b583600c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.021301 2377 rpcz_store.cc:276] 1026 02:56:21.935685 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:21.939543 (+ 3858us) service_pool.cc:225] Handling call
1026 02:56:22.969385 (+1029842us) ts_tablet_manager.cc:1151] Deleting tablet ac43df5558634550a12aa65c521d8a14
1026 02:56:22.969396 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:23.013076 (+ 43680us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:23.017991 (+ 4915us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:23.020689 (+ 2698us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029622}
I20251026 02:56:23.022840 2377 tablet_replica.cc:333] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.023204 2377 raft_consensus.cc:2243] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.023456 2377 raft_consensus.cc:2272] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.024068 2377 ts_tablet_manager.cc:1916] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.026901 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 155a23a93358487a9564bb5ac175b4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.031744 2377 ts_tablet_manager.cc:1929] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.031987 2377 log.cc:1199] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/5e56e5eec1474aefae04d278ded30765
I20251026 02:56:23.032465 2377 ts_tablet_manager.cc:1950] T 5e56e5eec1474aefae04d278ded30765 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.032951 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 20dec74440cf4bb9949c462ec51e9889 with delete_type TABLET_DATA_DELETED (replaced by tablet c570e5f9a93c45bda32ec92c68794714) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.033776 2377 tablet_replica.cc:333] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.034047 2377 raft_consensus.cc:2243] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:23.034400 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet ac43df5558634550a12aa65c521d8a14 (table default.Table_8827f142daf64c45b7786c00d72ace16 [id=c881f3e0b92f45788e52269792793f47]) successfully deleted
I20251026 02:56:23.034494 2377 raft_consensus.cc:2272] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.034955 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 5e56e5eec1474aefae04d278ded30765 (table default.Table_83bdf2cfef4042748553ebe8ae34996a [id=6734e5731cc94e71b43c76512f649d57]) successfully deleted
I20251026 02:56:23.035290 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 20dec74440cf4bb9949c462ec51e9889 with delete_type TABLET_DATA_DELETED (replaced by tablet c570e5f9a93c45bda32ec92c68794714) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:23.035614 2377 ts_tablet_manager.cc:1916] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:22.962184 2375 ts_tablet_manager.cc:1403] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.002s
I20251026 02:56:23.039026 2372 log.cc:1199] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3fd66abba7364951a7659011c71e76af
I20251026 02:56:23.039498 2372 ts_tablet_manager.cc:1950] T 3fd66abba7364951a7659011c71e76af P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.039805 2375 raft_consensus.cc:359] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.040037 2375 raft_consensus.cc:385] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.040103 2375 raft_consensus.cc:740] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.040520 2375 consensus_queue.cc:260] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.040940 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3fd66abba7364951a7659011c71e76af (table default.Table_99b42c4a4a874b68bb4beba4c08cdeb0 [id=472d44e7e6f0409ab9d238c4913cc8a8]) successfully deleted
I20251026 02:56:23.041064 2375 ts_tablet_manager.cc:1434] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:23.041369 2372 tablet_replica.cc:333] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.041471 2375 tablet_bootstrap.cc:492] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.041592 2372 raft_consensus.cc:2243] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:23.042009 2372 pending_rounds.cc:70] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48: Trying to abort 1 pending ops.
I20251026 02:56:23.042373 2372 pending_rounds.cc:77] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888478864756736 op_type: NO_OP noop_request { }
I20251026 02:56:23.042660 2372 raft_consensus.cc:2889] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:23.042788 2372 raft_consensus.cc:2272] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.043267 2375 tablet_bootstrap.cc:654] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.044946 2376 log.cc:1199] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/5e56e5eec1474aefae04d278ded30765
I20251026 02:56:22.964632 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 9a08e8b7eb1949408f2151f2d7257e9d (DEFAULT_TABLE table=default.Table_a5ff6167974b47798ffe2b3cd583b50b [id=43cb7f538f824ae38059645230cc7e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.046178 2376 ts_tablet_manager.cc:1950] T 5e56e5eec1474aefae04d278ded30765 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:23.046250 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a08e8b7eb1949408f2151f2d7257e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:22.966624 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 340bf8c78a88413299b61a52f0a578f3 (DEFAULT_TABLE table=default.Table_627cb33ef7ce4082b858fa0670d09862 [id=c162ef89d2f14e91a24a6d8c2c37c261]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.046865 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340bf8c78a88413299b61a52f0a578f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.047083 2376 tablet_replica.cc:333] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.047330 2376 raft_consensus.cc:2243] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:23.047809 2376 raft_consensus.cc:2272] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.017242 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 20dec74440cf4bb9949c462ec51e9889 with delete_type TABLET_DATA_DELETED (replaced by tablet c570e5f9a93c45bda32ec92c68794714) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.050191 2376 ts_tablet_manager.cc:1916] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.060094 2378 tablet_bootstrap.cc:492] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:23.060284 2378 ts_tablet_manager.cc:1403] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.107s user 0.004s sys 0.000s
I20251026 02:56:23.061087 2378 raft_consensus.cc:359] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.061305 2378 raft_consensus.cc:385] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.061411 2378 raft_consensus.cc:740] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.061646 2378 consensus_queue.cc:260] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.062186 2378 ts_tablet_manager.cc:1434] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:23.062453 2378 tablet_bootstrap.cc:492] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:23.062664 2381 tablet_bootstrap.cc:492] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:23.062814 2381 ts_tablet_manager.cc:1403] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.076s user 0.018s sys 0.001s
I20251026 02:56:23.063555 2381 raft_consensus.cc:359] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.063723 2381 raft_consensus.cc:385] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.063819 2381 raft_consensus.cc:740] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.065325 2378 tablet_bootstrap.cc:654] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.062635 2372 ts_tablet_manager.cc:1916] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.066223 2381 consensus_queue.cc:260] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.066761 2381 ts_tablet_manager.cc:1434] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:23.069677 2381 tablet_bootstrap.cc:492] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:23.069761 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5cda8845fc8f42e9a04ed877e65b2be3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.070344 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cda8845fc8f42e9a04ed877e65b2be3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.071393 2381 tablet_bootstrap.cc:654] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.071745 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fa379fbb05ce4e46a08b7fc1bae3183a" } modify_external_catalogs: true
I20251026 02:56:23.071957 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fa379fbb05ce4e46a08b7fc1bae3183a" } modify_external_catalogs: true
I20251026 02:56:23.078673 2377 ts_tablet_manager.cc:1929] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:23.080178 2377 log.cc:1199] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/760e6c761a7b49f695308fa1da9eda37
I20251026 02:56:23.081116 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e56e5eec1474aefae04d278ded30765 with delete_type TABLET_DATA_DELETED (replaced by tablet 23fd1284c9cb41bd98209ccbfe342bca) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.081514 2377 ts_tablet_manager.cc:1950] T 760e6c761a7b49f695308fa1da9eda37 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.082531 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 91) took 1017 ms. Trace:
I20251026 02:56:23.084153 2377 rpcz_store.cc:276] 1026 02:56:22.064777 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:22.065478 (+ 701us) service_pool.cc:225] Handling call
1026 02:56:23.033685 (+968207us) ts_tablet_manager.cc:1151] Deleting tablet 760e6c761a7b49f695308fa1da9eda37
1026 02:56:23.033694 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:23.059940 (+ 26246us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:23.078510 (+ 18570us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:23.082495 (+ 3985us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":924059}
I20251026 02:56:23.085562 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 90) took 1021 ms. Trace:
I20251026 02:56:23.086356 2377 rpcz_store.cc:276] 1026 02:56:22.064593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:22.065670 (+ 1077us) service_pool.cc:225] Handling call
1026 02:56:23.085388 (+1019718us) ts_tablet_manager.cc:1151] Deleting tablet 2ee0a408253e4f2b942c510ae64c703e
1026 02:56:23.085400 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:23.085517 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":975700}
I20251026 02:56:23.088251 2377 tablet_replica.cc:333] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.088917 2377 raft_consensus.cc:2243] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.089695 2377 raft_consensus.cc:2272] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.090682 2377 ts_tablet_manager.cc:1916] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.097304 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 565255802f4f42ba8320a3ec80d2db8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:23.097834 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 340bf8c78a88413299b61a52f0a578f3 (DEFAULT_TABLE table=default.Table_627cb33ef7ce4082b858fa0670d09862 [id=c162ef89d2f14e91a24a6d8c2c37c261]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.098414 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340bf8c78a88413299b61a52f0a578f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.098748 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 760e6c761a7b49f695308fa1da9eda37 (table default.Table_6b07e3e332c141aa824ea90a76f40208 [id=c3e9c585b810403f88e4eb2b4924b73c]) successfully deleted
W20251026 02:56:23.099035 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 2ee0a408253e4f2b942c510ae64c703e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ee0a408253e4f2b942c510ae64c703e already in progress: creating tablet
I20251026 02:56:23.099182 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a04ac20b7624a018c403ef4f64f7a69 with delete_type TABLET_DATA_DELETED (replaced by tablet 340bf8c78a88413299b61a52f0a578f3) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:23.099293 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ad55040f0a2b4cfa894d1bd058d4d69f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad55040f0a2b4cfa894d1bd058d4d69f already in progress: creating tablet
I20251026 02:56:23.099714 2375 tablet_bootstrap.cc:492] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.099869 2375 ts_tablet_manager.cc:1403] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.007s
I20251026 02:56:23.101657 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7814874203fc40af9ef07eeb2e11ebc4" } modify_external_catalogs: true
I20251026 02:56:23.101745 2375 raft_consensus.cc:359] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.101862 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7814874203fc40af9ef07eeb2e11ebc4" } modify_external_catalogs: true
I20251026 02:56:23.101987 2375 raft_consensus.cc:385] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.102087 2375 raft_consensus.cc:740] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.102382 2375 consensus_queue.cc:260] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.102504 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_66dd770512e847f4a14ee04e5afa3a6c" } modify_external_catalogs: true
I20251026 02:56:23.102656 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_66dd770512e847f4a14ee04e5afa3a6c" } modify_external_catalogs: true
I20251026 02:56:23.095773 2376 ts_tablet_manager.cc:1929] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:23.097698 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fe6cbe999c54b299d729e97f0f938e7 with delete_type TABLET_DATA_DELETED (replaced by tablet ad55040f0a2b4cfa894d1bd058d4d69f) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.104501 2376 log.cc:1199] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/82315e4c5a554eeca5048a496498ca1f
I20251026 02:56:23.105067 2376 ts_tablet_manager.cc:1950] T 82315e4c5a554eeca5048a496498ca1f P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:23.105062 2375 ts_tablet_manager.cc:1434] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251026 02:56:23.106024 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4c63b6bed4b1437f9a11cba9c5d09d15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:23.106274 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 5e56e5eec1474aefae04d278ded30765 (table default.Table_83bdf2cfef4042748553ebe8ae34996a [id=6734e5731cc94e71b43c76512f649d57]) successfully deleted
I20251026 02:56:23.106323 2375 tablet_bootstrap.cc:492] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.106382 2376 tablet_replica.cc:333] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.106568 2376 raft_consensus.cc:2243] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:23.106679 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c63b6bed4b1437f9a11cba9c5d09d15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.106715 2376 raft_consensus.cc:2272] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.106731 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 82315e4c5a554eeca5048a496498ca1f (table default.Table_cfeac8daaf08492a9456d2e06cd62a58 [id=1d09f5b9fa164ac783f0be9332eac4c3]) successfully deleted
I20251026 02:56:23.107996 2375 tablet_bootstrap.cc:654] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.109220 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet ea910e98d17340efb816a8aa90f0ba7e from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:23.111011 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e071f7a8a409464d810f12e647a4ff0e" } modify_external_catalogs: true
I20251026 02:56:23.111219 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e071f7a8a409464d810f12e647a4ff0e" } modify_external_catalogs: true
I20251026 02:56:23.111794 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eea9b18e5094a22a51942f438a20eb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.112874 2376 ts_tablet_manager.cc:1916] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.117182 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 2487bf7d40f34922aa2fd165488342f6 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:23.119045 2381 tablet_bootstrap.cc:492] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:23.119342 2381 ts_tablet_manager.cc:1403] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.006s
I20251026 02:56:23.120407 2381 raft_consensus.cc:359] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.120712 2381 raft_consensus.cc:385] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.120852 2381 raft_consensus.cc:740] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.121230 2381 consensus_queue.cc:260] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.122988 2375 tablet_bootstrap.cc:492] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.123162 2375 ts_tablet_manager.cc:1403] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20251026 02:56:23.123765 2381 ts_tablet_manager.cc:1434] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:23.123976 2375 raft_consensus.cc:359] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.124186 2375 raft_consensus.cc:385] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.124289 2375 raft_consensus.cc:740] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.124826 2375 consensus_queue.cc:260] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.127065 2375 ts_tablet_manager.cc:1434] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:23.127568 2375 tablet_bootstrap.cc:492] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.129347 2372 ts_tablet_manager.cc:1929] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:23.121345 2377 ts_tablet_manager.cc:1929] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.130368 2377 log.cc:1199] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/e79223b1086347199470a2526e32bd6f
I20251026 02:56:23.130839 2377 ts_tablet_manager.cc:1950] T e79223b1086347199470a2526e32bd6f P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.132824 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 93) took 1022 ms. Trace:
I20251026 02:56:23.133216 2377 rpcz_store.cc:276] 1026 02:56:22.110656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:22.114224 (+ 3568us) service_pool.cc:225] Handling call
1026 02:56:23.088175 (+973951us) ts_tablet_manager.cc:1151] Deleting tablet e79223b1086347199470a2526e32bd6f
1026 02:56:23.088184 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:23.095865 (+ 7681us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:23.121219 (+ 25354us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:23.132767 (+ 11548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":973773}
I20251026 02:56:23.133701 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet e79223b1086347199470a2526e32bd6f (table default.Table_382e6508ce4645699117b0e65595251c [id=214ae71fbaf24f4ab016cb643a89f97f]) successfully deleted
I20251026 02:56:23.133800 2377 tablet_replica.cc:333] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.134089 2377 raft_consensus.cc:2243] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.134334 2377 raft_consensus.cc:2272] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.135145 2377 ts_tablet_manager.cc:1916] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.135277 2376 ts_tablet_manager.cc:1929] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.135427 2376 log.cc:1199] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/4eea9b18e5094a22a51942f438a20eb4
I20251026 02:56:23.135560 2378 tablet_bootstrap.cc:492] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:23.135730 2378 ts_tablet_manager.cc:1403] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.007s
I20251026 02:56:23.135809 2376 ts_tablet_manager.cc:1950] T 4eea9b18e5094a22a51942f438a20eb4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:23.137001 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 4eea9b18e5094a22a51942f438a20eb4 (table default.Table_71bb5e64488b4f4f92c57f6a1e34fb3c [id=30a69f019af84e09af9f21939f081847]) successfully deleted
W20251026 02:56:23.137672 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7036a159e88b4171942dd238f8af3ed3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7036a159e88b4171942dd238f8af3ed3 already in progress: creating tablet
I20251026 02:56:23.139971 2376 tablet_replica.cc:333] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.140241 2376 raft_consensus.cc:2243] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.140465 2376 raft_consensus.cc:2272] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.141661 2376 ts_tablet_manager.cc:1916] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.141906 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet b108f0d3dae84fab93bff012400c4a2a from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:23.142495 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d92e0992e0274b39adde1edd34491399"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:23.142807 2377 ts_tablet_manager.cc:1929] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.142926 2377 log.cc:1199] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/426ce6fb80874df68210a652f7b7602f
W20251026 02:56:23.143065 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d92e0992e0274b39adde1edd34491399 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.143307 2377 ts_tablet_manager.cc:1950] T 426ce6fb80874df68210a652f7b7602f P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.146649 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 95) took 1035 ms. Trace:
I20251026 02:56:23.146826 2377 rpcz_store.cc:276] 1026 02:56:22.110894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:22.114438 (+ 3544us) service_pool.cc:225] Handling call
1026 02:56:23.133708 (+1019270us) ts_tablet_manager.cc:1151] Deleting tablet 426ce6fb80874df68210a652f7b7602f
1026 02:56:23.133719 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:23.139093 (+ 5374us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:23.142666 (+ 3573us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:23.146604 (+ 3938us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1019098}
W20251026 02:56:23.147244 1916 consensus_peers.cc:597] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:23.147419 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 426ce6fb80874df68210a652f7b7602f (table default.Table_ff61ac36382f428598c86ecb81001e88 [id=6071000b0280484b8be3da135c7f7b74]) successfully deleted
I20251026 02:56:23.147739 2377 tablet_replica.cc:333] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.147994 2377 raft_consensus.cc:2243] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.148386 2377 raft_consensus.cc:2272] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.148468 2381 tablet_bootstrap.cc:492] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:23.149036 2377 ts_tablet_manager.cc:1916] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.150305 2381 tablet_bootstrap.cc:654] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.152678 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b108f0d3dae84fab93bff012400c4a2a
I20251026 02:56:23.153607 2378 raft_consensus.cc:359] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.153861 2378 raft_consensus.cc:385] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.153963 2378 raft_consensus.cc:740] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.154423 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet b108f0d3dae84fab93bff012400c4a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:23.154700 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet b108f0d3dae84fab93bff012400c4a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.155110 2378 consensus_queue.cc:260] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.155694 2376 ts_tablet_manager.cc:1929] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.155807 2376 log.cc:1199] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/79942752cba4429abdc9524f882800da
I20251026 02:56:23.156177 2376 ts_tablet_manager.cc:1950] T 79942752cba4429abdc9524f882800da P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:23.156296 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_90929641047f40f1973c0e399816b5ef" } modify_external_catalogs: true
I20251026 02:56:23.156490 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_90929641047f40f1973c0e399816b5ef" } modify_external_catalogs: true
I20251026 02:56:23.133140 2375 tablet_bootstrap.cc:654] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.157196 2372 log.cc:1199] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/20c39a62bfab429bb231ae073585d0e5
I20251026 02:56:23.157764 2372 ts_tablet_manager.cc:1950] T 20c39a62bfab429bb231ae073585d0e5 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.156976 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet b108f0d3dae84fab93bff012400c4a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.145871 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 79942752cba4429abdc9524f882800da with delete_type TABLET_DATA_DELETED (replaced by tablet 9a08e8b7eb1949408f2151f2d7257e9d) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.160913 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 20c39a62bfab429bb231ae073585d0e5 (table default.Table_0aa77747f2be46d8be9e7c5873940ca1 [id=a414f870eb244e0ba5d2569c6c91edc0]) successfully deleted
W20251026 02:56:23.161718 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 already in progress: creating tablet
I20251026 02:56:23.162222 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b435520cadb344edaa29975b4e19718a
I20251026 02:56:23.162338 2372 tablet_replica.cc:333] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.162518 2372 raft_consensus.cc:2243] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.162853 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 0020cb1f190c49978c5322d387a47f71 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:23.163762 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet b435520cadb344edaa29975b4e19718a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.164026 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet b435520cadb344edaa29975b4e19718a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.164086 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet b435520cadb344edaa29975b4e19718a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:23.162711 2372 raft_consensus.cc:2272] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.164871 2372 ts_tablet_manager.cc:1916] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.166312 2377 ts_tablet_manager.cc:1929] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.166496 2377 log.cc:1199] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7fe6cbe999c54b299d729e97f0f938e7
I20251026 02:56:23.166471 2378 ts_tablet_manager.cc:1434] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20251026 02:56:23.166818 2378 tablet_bootstrap.cc:492] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:23.166956 2377 ts_tablet_manager.cc:1950] T 7fe6cbe999c54b299d729e97f0f938e7 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.166913 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 79942752cba4429abdc9524f882800da (table default.Table_a5ff6167974b47798ffe2b3cd583b50b [id=43cb7f538f824ae38059645230cc7e94]) successfully deleted
W20251026 02:56:23.167701 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet fd1484434dd64f4bb2764268b3d3b218 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1484434dd64f4bb2764268b3d3b218 already in progress: creating tablet
I20251026 02:56:23.167761 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 94) took 1056 ms. Trace:
I20251026 02:56:23.167989 2377 rpcz_store.cc:276] 1026 02:56:22.110789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:22.116326 (+ 5537us) service_pool.cc:225] Handling call
1026 02:56:23.147662 (+1031336us) ts_tablet_manager.cc:1151] Deleting tablet 7fe6cbe999c54b299d729e97f0f938e7
1026 02:56:23.147673 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:23.152886 (+ 5213us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:23.166179 (+ 13293us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:23.167705 (+ 1526us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1031043}
I20251026 02:56:23.168036 2376 tablet_replica.cc:333] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.168578 2376 raft_consensus.cc:2243] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.168691 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7fe6cbe999c54b299d729e97f0f938e7 (table default.Table_0da51595453549b3b7e077f7b7ff3ec7 [id=53cd517e68e64597ad51d85d16f0399d]) successfully deleted
I20251026 02:56:23.168818 2376 raft_consensus.cc:2272] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.168949 2405 raft_consensus.cc:493] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:23.169095 2377 tablet_replica.cc:333] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.169111 2405 raft_consensus.cc:515] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.169816 2375 tablet_bootstrap.cc:492] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.169991 2375 ts_tablet_manager.cc:1403] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.007s
I20251026 02:56:23.170291 2405 leader_election.cc:290] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:23.170733 2375 raft_consensus.cc:359] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.170920 2375 raft_consensus.cc:385] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.170992 2375 raft_consensus.cc:740] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.171577 2376 ts_tablet_manager.cc:1916] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.172474 2375 consensus_queue.cc:260] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.172616 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ef7247d62ec4adc82fdbed40fc852ea" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:23.174618 2195 leader_election.cc:343] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:23.172154 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ef7247d62ec4adc82fdbed40fc852ea" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:23.175406 2194 leader_election.cc:343] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:23.175523 2194 leader_election.cc:304] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:23.176409 2405 raft_consensus.cc:2749] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:23.178261 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 426ce6fb80874df68210a652f7b7602f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.178850 2375 ts_tablet_manager.cc:1434] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20251026 02:56:23.180039 2375 tablet_bootstrap.cc:492] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.182601 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet e79223b1086347199470a2526e32bd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.183152 2375 tablet_bootstrap.cc:654] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.184247 2377 raft_consensus.cc:2243] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.184561 2377 raft_consensus.cc:2272] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.185422 2377 ts_tablet_manager.cc:1916] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.186934 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ac4b46f3e31e4929bc07664386d206f2" } new_table_name: "default.Table_de29df31fe874815a8f1b1d25f4ba210" modify_external_catalogs: true
I20251026 02:56:23.189379 2376 ts_tablet_manager.cc:1929] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.192932 2376 log.cc:1199] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/9a04ac20b7624a018c403ef4f64f7a69
I20251026 02:56:23.196823 2376 ts_tablet_manager.cc:1950] T 9a04ac20b7624a018c403ef4f64f7a69 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:23.197286 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ea910e98d17340efb816a8aa90f0ba7e
I20251026 02:56:23.190260 2378 tablet_bootstrap.cc:654] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.197863 1655 catalog_manager.cc:6719] ReplaceTablet: tablet ea910e98d17340efb816a8aa90f0ba7e deleted and replaced by tablet 1370b9ad917344c093e5da3d0e9b1cf0
I20251026 02:56:23.198554 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet ea910e98d17340efb816a8aa90f0ba7e with delete_type TABLET_DATA_DELETED (replaced by tablet 1370b9ad917344c093e5da3d0e9b1cf0) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.198801 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet ea910e98d17340efb816a8aa90f0ba7e with delete_type TABLET_DATA_DELETED (replaced by tablet 1370b9ad917344c093e5da3d0e9b1cf0) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:23.200423 2378 tablet_bootstrap.cc:492] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:23.200572 2378 ts_tablet_manager.cc:1403] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.002s
I20251026 02:56:23.200925 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 9a04ac20b7624a018c403ef4f64f7a69 (table default.Table_627cb33ef7ce4082b858fa0670d09862 [id=c162ef89d2f14e91a24a6d8c2c37c261]) successfully deleted
I20251026 02:56:23.201470 2376 tablet_replica.cc:333] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.201392 2378 raft_consensus.cc:359] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.201584 2378 raft_consensus.cc:385] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.201696 2378 raft_consensus.cc:740] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.201960 2378 consensus_queue.cc:260] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.201704 2376 raft_consensus.cc:2243] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:23.203068 2376 pending_rounds.cc:70] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8: Trying to abort 1 pending ops.
I20251026 02:56:23.203146 2376 pending_rounds.cc:77] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888480836464640 op_type: NO_OP noop_request { }
I20251026 02:56:23.205896 2377 ts_tablet_manager.cc:1929] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:23.203280 2376 raft_consensus.cc:2889] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:23.206460 2376 raft_consensus.cc:2272] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.207435 2377 log.cc:1199] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/82315e4c5a554eeca5048a496498ca1f
I20251026 02:56:23.208088 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet ea910e98d17340efb816a8aa90f0ba7e with delete_type TABLET_DATA_DELETED (replaced by tablet 1370b9ad917344c093e5da3d0e9b1cf0) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.208326 2377 ts_tablet_manager.cc:1950] T 82315e4c5a554eeca5048a496498ca1f P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.209254 2375 tablet_bootstrap.cc:492] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.209501 2375 ts_tablet_manager.cc:1403] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.007s
I20251026 02:56:23.203666 2378 ts_tablet_manager.cc:1434] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:23.210140 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ff86fc06e7d46e78fe68cc6f31a30cc with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.210788 2375 raft_consensus.cc:359] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.211077 2375 raft_consensus.cc:385] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.211205 2375 raft_consensus.cc:740] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.211516 2375 consensus_queue.cc:260] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.213157 2378 tablet_bootstrap.cc:492] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:23.213618 2506 raft_consensus.cc:493] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:23.213748 2506 raft_consensus.cc:515] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.214347 2372 ts_tablet_manager.cc:1929] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.214387 2506 leader_election.cc:290] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:23.214481 2372 log.cc:1199] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/64d00ae95e14439eaa748d17ca67118a
I20251026 02:56:23.214861 2372 ts_tablet_manager.cc:1950] T 64d00ae95e14439eaa748d17ca67118a P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.214977 2378 tablet_bootstrap.cc:654] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.216140 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 64d00ae95e14439eaa748d17ca67118a (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
W20251026 02:56:23.216444 2053 leader_election.cc:343] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 3802bec60e074107b9a8f8ddc3895fe2
W20251026 02:56:23.216627 2053 leader_election.cc:343] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 3802bec60e074107b9a8f8ddc3895fe2
I20251026 02:56:23.216717 2053 leader_election.cc:304] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:23.217180 2506 raft_consensus.cc:2749] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:23.217621 2375 ts_tablet_manager.cc:1434] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251026 02:56:23.218194 2372 tablet_replica.cc:333] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.218200 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c32c2393a1144e9950b00f61ddba4eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.218410 2372 raft_consensus.cc:2243] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.218580 2372 raft_consensus.cc:2272] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.219138 2372 ts_tablet_manager.cc:1916] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.222573 2375 tablet_bootstrap.cc:492] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.223242 2376 ts_tablet_manager.cc:1916] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.224201 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 82315e4c5a554eeca5048a496498ca1f (table default.Table_cfeac8daaf08492a9456d2e06cd62a58 [id=1d09f5b9fa164ac783f0be9332eac4c3]) successfully deleted
I20251026 02:56:23.225247 2375 tablet_bootstrap.cc:654] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:23.225863 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7036a159e88b4171942dd238f8af3ed3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7036a159e88b4171942dd238f8af3ed3 already in progress: creating tablet
I20251026 02:56:23.226387 2377 tablet_replica.cc:333] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.226668 2377 raft_consensus.cc:2243] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.228286 2377 raft_consensus.cc:2272] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.228957 2377 ts_tablet_manager.cc:1916] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.229559 2381 tablet_bootstrap.cc:492] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:23.229727 2381 ts_tablet_manager.cc:1403] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.004s
I20251026 02:56:23.230607 2381 raft_consensus.cc:359] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.230798 2381 raft_consensus.cc:385] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.230906 2381 raft_consensus.cc:740] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.231174 2381 consensus_queue.cc:260] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.231861 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_698ae47452154028aa91996591984443" } new_table_name: "default.Table_d23ac2c9657b4adeaac9545d5bc9ae41" modify_external_catalogs: true
I20251026 02:56:23.232995 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0ffb653ff26d42cdb0276a492f332424" } new_table_name: "default.Table_6546379855b749b889279973db1820cd" modify_external_catalogs: true
W20251026 02:56:23.236688 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 903e65fa8c214dc9a50cb97cab101d22 has no consensus state
I20251026 02:56:23.237784 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_cba8c65ea6a14b09a43a61d89e0ecee4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.238597 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cba8c65ea6a14b09a43a61d89e0ecee4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.240725 2377 ts_tablet_manager.cc:1929] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.242678 2376 ts_tablet_manager.cc:1929] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:23.242851 2376 log.cc:1199] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/7c32c2393a1144e9950b00f61ddba4eb
I20251026 02:56:23.243332 2376 ts_tablet_manager.cc:1950] T 7c32c2393a1144e9950b00f61ddba4eb P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:23.243001 2381 ts_tablet_manager.cc:1434] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251026 02:56:23.244266 2377 log.cc:1199] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/4eea9b18e5094a22a51942f438a20eb4
I20251026 02:56:23.245187 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 7c32c2393a1144e9950b00f61ddba4eb (table default.Table_d89d54d7dc874d7b8d7599cbe59b281a [id=85ee8f90981b4b4f899a45b6c29ff1e0]) successfully deleted
W20251026 02:56:23.245472 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 already in progress: creating tablet
I20251026 02:56:23.245963 2372 ts_tablet_manager.cc:1929] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.246193 2372 log.cc:1199] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/19edc0546fd04110b37353178da2652e
I20251026 02:56:23.246193 2377 ts_tablet_manager.cc:1950] T 4eea9b18e5094a22a51942f438a20eb4 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.246582 2372 ts_tablet_manager.cc:1950] T 19edc0546fd04110b37353178da2652e P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.247160 2377 tablet_replica.cc:333] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.247594 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 4eea9b18e5094a22a51942f438a20eb4 (table default.Table_71bb5e64488b4f4f92c57f6a1e34fb3c [id=30a69f019af84e09af9f21939f081847]) successfully deleted
I20251026 02:56:23.247706 2372 tablet_replica.cc:333] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.247977 2372 raft_consensus.cc:2243] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.248332 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 19edc0546fd04110b37353178da2652e (table default.Table_f26f32ab73e848fea8718bfddbebb31f [id=aa6a3819787c4ab881cb8d71460a9b09]) successfully deleted
I20251026 02:56:23.248263 2372 raft_consensus.cc:2272] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.251169 2372 ts_tablet_manager.cc:1916] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.252964 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]). Expected version 2 got 0
I20251026 02:56:23.249239 2381 tablet_bootstrap.cc:492] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:23.254813 2376 tablet_replica.cc:333] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.255098 2376 raft_consensus.cc:2243] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:23.255483 2376 pending_rounds.cc:70] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8: Trying to abort 1 pending ops.
I20251026 02:56:23.255610 2376 pending_rounds.cc:77] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888477257916416 op_type: NO_OP noop_request { }
I20251026 02:56:23.255801 2376 raft_consensus.cc:2889] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:23.255967 2376 raft_consensus.cc:2272] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.256618 2381 tablet_bootstrap.cc:654] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.256347 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 (DEFAULT_TABLE table=default.Table_7296727c5b2f4c95ac3ff62818bd84f2 [id=af7a0605122745588d028814d3583cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.256633 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 (DEFAULT_TABLE table=default.Table_7296727c5b2f4c95ac3ff62818bd84f2 [id=af7a0605122745588d028814d3583cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.257061 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a5343ba771a4e8ea2f467c4c619c1f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.257055 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a5343ba771a4e8ea2f467c4c619c1f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.257381 2376 ts_tablet_manager.cc:1916] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.258304 2377 raft_consensus.cc:2243] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.258509 2377 raft_consensus.cc:2272] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.259028 2377 ts_tablet_manager.cc:1916] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.265828 2381 tablet_bootstrap.cc:492] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:23.266033 2381 ts_tablet_manager.cc:1403] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.001s
I20251026 02:56:23.266149 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 (DEFAULT_TABLE table=default.Table_7296727c5b2f4c95ac3ff62818bd84f2 [id=af7a0605122745588d028814d3583cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.266589 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a5343ba771a4e8ea2f467c4c619c1f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.266676 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 903e65fa8c214dc9a50cb97cab101d22 (DEFAULT_TABLE table=default.Table_de29df31fe874815a8f1b1d25f4ba210 [id=65b5770c4b9243b892bfab963a439bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.266875 2381 raft_consensus.cc:359] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.267081 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 903e65fa8c214dc9a50cb97cab101d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.267100 2381 raft_consensus.cc:385] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.267182 2381 raft_consensus.cc:740] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.272454 2381 consensus_queue.cc:260] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.273577 2372 ts_tablet_manager.cc:1929] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.275794 2381 ts_tablet_manager.cc:1434] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251026 02:56:23.276285 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 903e65fa8c214dc9a50cb97cab101d22 (DEFAULT_TABLE table=default.Table_de29df31fe874815a8f1b1d25f4ba210 [id=65b5770c4b9243b892bfab963a439bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.276752 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 903e65fa8c214dc9a50cb97cab101d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.276757 2381 tablet_bootstrap.cc:492] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:23.278756 2381 tablet_bootstrap.cc:654] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.275338 2506 raft_consensus.cc:493] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:23.279621 2372 log.cc:1199] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3802bec60e074107b9a8f8ddc3895fe2
I20251026 02:56:23.279649 2506 raft_consensus.cc:515] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.280071 2372 ts_tablet_manager.cc:1950] T 3802bec60e074107b9a8f8ddc3895fe2 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.280447 2506 leader_election.cc:290] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:23.280520 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 903e65fa8c214dc9a50cb97cab101d22 (DEFAULT_TABLE table=default.Table_de29df31fe874815a8f1b1d25f4ba210 [id=65b5770c4b9243b892bfab963a439bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.281139 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a04ac20b7624a018c403ef4f64f7a69" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:23.281373 2262 raft_consensus.cc:2468] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:23.281648 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3802bec60e074107b9a8f8ddc3895fe2 (table default.Table_1b8997fc8d6b4f788983c76e84b2d0c5 [id=37382ab63e7d448788deaa4cd95c8b44]) successfully deleted
I20251026 02:56:23.282018 2053 leader_election.cc:304] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
W20251026 02:56:23.282346 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 1c1596dd4394419a8bb6ea6a84be9fea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c1596dd4394419a8bb6ea6a84be9fea already in progress: creating tablet
I20251026 02:56:23.282410 2506 raft_consensus.cc:2804] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:23.282536 2506 raft_consensus.cc:493] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 02:56:23.282594 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 461412f3ad1e4186b1302cdf5a624ed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 461412f3ad1e4186b1302cdf5a624ed4 already in progress: creating tablet
I20251026 02:56:23.282632 2506 raft_consensus.cc:3060] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:23.282804 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet e0101098df05462bb26b3c187b259c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0101098df05462bb26b3c187b259c08 already in progress: creating tablet
I20251026 02:56:23.282909 2372 tablet_replica.cc:333] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:23.283035 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ac19c5a847454a23ab16831198ff2a20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac19c5a847454a23ab16831198ff2a20 already in progress: creating tablet
I20251026 02:56:23.283226 2372 raft_consensus.cc:2243] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.283445 2372 raft_consensus.cc:2272] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.280566 2376 ts_tablet_manager.cc:1929] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:23.283797 2376 log.cc:1199] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/e79223b1086347199470a2526e32bd6f
W20251026 02:56:23.283881 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f60bcc5e4cb4fd19d2a5713f69814cd already in progress: creating tablet
W20251026 02:56:23.284106 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0e89c69e3b58458aad10beaeb5128d60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e89c69e3b58458aad10beaeb5128d60 already in progress: creating tablet
I20251026 02:56:23.284289 2376 ts_tablet_manager.cc:1950] T e79223b1086347199470a2526e32bd6f P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
W20251026 02:56:23.284349 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet c0f83d631faa468f828e54e97987a00b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0f83d631faa468f828e54e97987a00b already in progress: creating tablet
I20251026 02:56:23.284435 2372 ts_tablet_manager.cc:1916] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:23.284541 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 20dec74440cf4bb9949c462ec51e9889 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20dec74440cf4bb9949c462ec51e9889 already in progress: creating tablet
I20251026 02:56:23.285808 2506 raft_consensus.cc:515] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.287587 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a04ac20b7624a018c403ef4f64f7a69" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:23.287806 2262 raft_consensus.cc:3060] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:23.289676 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 7845ebd3e9ca4897b8fb19a48a010315 (DEFAULT_TABLE table=default.Table_9c97bccbf4cb4d15a66dbb8130acb6d4 [id=10d722da66cd4661a1bb1ab82997dca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.289757 2377 ts_tablet_manager.cc:1929] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.289875 2377 log.cc:1199] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/79942752cba4429abdc9524f882800da
I20251026 02:56:23.290305 2377 ts_tablet_manager.cc:1950] T 79942752cba4429abdc9524f882800da P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.290453 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 7845ebd3e9ca4897b8fb19a48a010315 (DEFAULT_TABLE table=default.Table_9c97bccbf4cb4d15a66dbb8130acb6d4 [id=10d722da66cd4661a1bb1ab82997dca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.290949 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7845ebd3e9ca4897b8fb19a48a010315. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.291078 2262 raft_consensus.cc:2468] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:23.291307 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 79942752cba4429abdc9524f882800da (table default.Table_a5ff6167974b47798ffe2b3cd583b50b [id=43cb7f538f824ae38059645230cc7e94]) successfully deleted
I20251026 02:56:23.291714 2053 leader_election.cc:304] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:23.292999 2506 leader_election.cc:290] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:23.293390 2506 raft_consensus.cc:2804] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:23.293771 2506 raft_consensus.cc:697] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:23.294263 2506 consensus_queue.cc:237] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.295259 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 8dbc12d49b094dbeadad2e76916ba9af (DEFAULT_TABLE table=default.Table_cdb10d198a394aac9033a98152fd2d94 [id=1d0784496733469fbff42783f90d56d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.295629 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d92e0992e0274b39adde1edd34491399" } new_table_name: "default.Table_7125b173069d4066a0b65f414c55c7e1" modify_external_catalogs: true
I20251026 02:56:23.297402 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5fdecebb943046c599c1f24317ce85f2" } new_table_name: "default.Table_54299d87a8e24d0484cc7f08d2b2e1f4" modify_external_catalogs: true
I20251026 02:56:23.300792 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 8dbc12d49b094dbeadad2e76916ba9af (DEFAULT_TABLE table=default.Table_cdb10d198a394aac9033a98152fd2d94 [id=1d0784496733469fbff42783f90d56d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.301337 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dbc12d49b094dbeadad2e76916ba9af. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:23.304103 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet fd1484434dd64f4bb2764268b3d3b218 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1484434dd64f4bb2764268b3d3b218 already in progress: creating tablet
I20251026 02:56:23.305195 2377 tablet_replica.cc:333] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.305428 2377 raft_consensus.cc:2243] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.305594 2377 raft_consensus.cc:2272] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.306198 2377 ts_tablet_manager.cc:1916] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.308624 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 7845ebd3e9ca4897b8fb19a48a010315 (DEFAULT_TABLE table=default.Table_9c97bccbf4cb4d15a66dbb8130acb6d4 [id=10d722da66cd4661a1bb1ab82997dca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.309048 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7845ebd3e9ca4897b8fb19a48a010315. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:23.309155 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 consensus state has no leader
I20251026 02:56:23.311837 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7845ebd3e9ca4897b8fb19a48a010315. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.312726 2372 ts_tablet_manager.cc:1929] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.312886 2372 log.cc:1199] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/5e56e5eec1474aefae04d278ded30765
I20251026 02:56:23.313311 2372 ts_tablet_manager.cc:1950] T 5e56e5eec1474aefae04d278ded30765 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.313657 1660 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]). Expected version 2 got 0
I20251026 02:56:23.313830 2376 tablet_replica.cc:333] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.314062 2376 raft_consensus.cc:2243] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.314276 2372 tablet_replica.cc:333] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.314260 2376 raft_consensus.cc:2272] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.314561 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet 1c1596dd4394419a8bb6ea6a84be9fea from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:23.314818 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dbc12d49b094dbeadad2e76916ba9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.315104 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_719b11bde46041c5a20750db14381807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.315665 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_719b11bde46041c5a20750db14381807 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.317025 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 5e56e5eec1474aefae04d278ded30765 (table default.Table_83bdf2cfef4042748553ebe8ae34996a [id=6734e5731cc94e71b43c76512f649d57]) successfully deleted
I20251026 02:56:23.318974 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 903e65fa8c214dc9a50cb97cab101d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.319082 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 8dbc12d49b094dbeadad2e76916ba9af (DEFAULT_TABLE table=default.Table_cdb10d198a394aac9033a98152fd2d94 [id=1d0784496733469fbff42783f90d56d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.319482 2090 tablet_service.cc:1505] Processing CreateTablet for tablet c570e5f9a93c45bda32ec92c68794714 (DEFAULT_TABLE table=default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.319630 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dbc12d49b094dbeadad2e76916ba9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.319895 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c570e5f9a93c45bda32ec92c68794714. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.320420 2405 raft_consensus.cc:493] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:23.320595 2405 raft_consensus.cc:515] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.322453 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "565255802f4f42ba8320a3ec80d2db8f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:23.322679 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "565255802f4f42ba8320a3ec80d2db8f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:23.323105 2194 leader_election.cc:343] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:23.323269 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet e79223b1086347199470a2526e32bd6f (table default.Table_382e6508ce4645699117b0e65595251c [id=214ae71fbaf24f4ab016cb643a89f97f]) successfully deleted
W20251026 02:56:23.323560 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 1c1596dd4394419a8bb6ea6a84be9fea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c1596dd4394419a8bb6ea6a84be9fea already in progress: creating tablet
W20251026 02:56:23.323786 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet e0101098df05462bb26b3c187b259c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0101098df05462bb26b3c187b259c08 already in progress: creating tablet
W20251026 02:56:23.323969 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 461412f3ad1e4186b1302cdf5a624ed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 461412f3ad1e4186b1302cdf5a624ed4 already in progress: creating tablet
W20251026 02:56:23.324185 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f60bcc5e4cb4fd19d2a5713f69814cd already in progress: creating tablet
W20251026 02:56:23.324381 2195 leader_election.cc:343] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:23.324445 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0e89c69e3b58458aad10beaeb5128d60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e89c69e3b58458aad10beaeb5128d60 already in progress: creating tablet
I20251026 02:56:23.324548 2195 leader_election.cc:304] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:23.324677 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ac19c5a847454a23ab16831198ff2a20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac19c5a847454a23ab16831198ff2a20 already in progress: creating tablet
I20251026 02:56:23.324903 2440 raft_consensus.cc:2749] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:23.324926 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet c0f83d631faa468f828e54e97987a00b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0f83d631faa468f828e54e97987a00b already in progress: creating tablet
W20251026 02:56:23.325155 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 20dec74440cf4bb9949c462ec51e9889 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20dec74440cf4bb9949c462ec51e9889 already in progress: creating tablet
I20251026 02:56:23.326177 2376 ts_tablet_manager.cc:1916] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.334517 1959 tablet_service.cc:1505] Processing CreateTablet for tablet c570e5f9a93c45bda32ec92c68794714 (DEFAULT_TABLE table=default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.335129 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c570e5f9a93c45bda32ec92c68794714. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:23.335870 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 829a6044a2da44ff90acdebca6d5b18d consensus state has no leader
I20251026 02:56:23.339937 2376 ts_tablet_manager.cc:1929] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.340122 2376 log.cc:1199] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/7fe6cbe999c54b299d729e97f0f938e7
I20251026 02:56:23.340289 2405 leader_election.cc:290] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:23.340523 2376 ts_tablet_manager.cc:1950] T 7fe6cbe999c54b299d729e97f0f938e7 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
W20251026 02:56:23.343847 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 52181.5us
I20251026 02:56:23.345044 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_828c16a96a47489f835a891f34d707b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.345484 2053 leader_election.cc:343] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 9a04ac20b7624a018c403ef4f64f7a69
W20251026 02:56:23.345682 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_828c16a96a47489f835a891f34d707b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.345952 2377 ts_tablet_manager.cc:1929] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251026 02:56:23.345979 2053 leader_election.cc:343] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 9a04ac20b7624a018c403ef4f64f7a69
I20251026 02:56:23.346100 2377 log.cc:1199] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/9a04ac20b7624a018c403ef4f64f7a69
I20251026 02:56:23.349696 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 7fe6cbe999c54b299d729e97f0f938e7 (table default.Table_0da51595453549b3b7e077f7b7ff3ec7 [id=53cd517e68e64597ad51d85d16f0399d]) successfully deleted
I20251026 02:56:23.349843 2376 tablet_replica.cc:333] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.350057 2376 raft_consensus.cc:2243] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.350222 2376 raft_consensus.cc:2272] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:23.350340 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b108f0d3dae84fab93bff012400c4a2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b108f0d3dae84fab93bff012400c4a2a already in progress: creating tablet
I20251026 02:56:23.350524 2377 ts_tablet_manager.cc:1950] T 9a04ac20b7624a018c403ef4f64f7a69 P 696476265495492790739ec126a08539: Deleting consensus metadata
W20251026 02:56:23.350603 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b435520cadb344edaa29975b4e19718a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b435520cadb344edaa29975b4e19718a already in progress: creating tablet
I20251026 02:56:23.351013 2376 ts_tablet_manager.cc:1916] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.351576 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 9a04ac20b7624a018c403ef4f64f7a69 (table default.Table_627cb33ef7ce4082b858fa0670d09862 [id=c162ef89d2f14e91a24a6d8c2c37c261]) successfully deleted
I20251026 02:56:23.351902 2377 tablet_replica.cc:333] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.352119 2377 raft_consensus.cc:2243] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.352339 2377 raft_consensus.cc:2272] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.352852 2377 ts_tablet_manager.cc:1916] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.358240 2231 tablet_service.cc:1505] Processing CreateTablet for tablet c570e5f9a93c45bda32ec92c68794714 (DEFAULT_TABLE table=default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.358453 1959 tablet_service.cc:1505] Processing CreateTablet for tablet a821ed8a686c452a975acb96b5c74778 (DEFAULT_TABLE table=default.Table_00dc4c49a19246889353df2157eebf15 [id=347d712aca4e477c89805846800fb824]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.358875 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a821ed8a686c452a975acb96b5c74778. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.358912 2090 tablet_service.cc:1505] Processing CreateTablet for tablet a821ed8a686c452a975acb96b5c74778 (DEFAULT_TABLE table=default.Table_00dc4c49a19246889353df2157eebf15 [id=347d712aca4e477c89805846800fb824]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.359452 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a821ed8a686c452a975acb96b5c74778. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.359764 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c570e5f9a93c45bda32ec92c68794714. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.368412 2376 ts_tablet_manager.cc:1929] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.368597 2376 log.cc:1199] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/426ce6fb80874df68210a652f7b7602f
I20251026 02:56:23.369014 2376 ts_tablet_manager.cc:1950] T 426ce6fb80874df68210a652f7b7602f P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:23.370028 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 426ce6fb80874df68210a652f7b7602f (table default.Table_ff61ac36382f428598c86ecb81001e88 [id=6071000b0280484b8be3da135c7f7b74]) successfully deleted
I20251026 02:56:23.370287 2372 raft_consensus.cc:2243] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:23.371210 2372 pending_rounds.cc:70] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48: Trying to abort 1 pending ops.
I20251026 02:56:23.371289 2372 pending_rounds.cc:77] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888481980231680 op_type: NO_OP noop_request { }
I20251026 02:56:23.371421 2372 raft_consensus.cc:2889] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:23.371526 2372 raft_consensus.cc:2272] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.373158 2377 ts_tablet_manager.cc:1929] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.373310 2377 log.cc:1199] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7c32c2393a1144e9950b00f61ddba4eb
I20251026 02:56:23.373474 2381 tablet_bootstrap.cc:492] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
W20251026 02:56:23.373555 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ea910e98d17340efb816a8aa90f0ba7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea910e98d17340efb816a8aa90f0ba7e already in progress: creating tablet
I20251026 02:56:23.373641 2381 ts_tablet_manager.cc:1403] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.098s user 0.001s sys 0.015s
I20251026 02:56:23.373736 2377 ts_tablet_manager.cc:1950] T 7c32c2393a1144e9950b00f61ddba4eb P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.373984 2376 tablet_replica.cc:333] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.374174 2376 raft_consensus.cc:2243] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.374370 2376 raft_consensus.cc:2272] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.374517 2381 raft_consensus.cc:359] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.374739 2381 raft_consensus.cc:385] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.374850 2381 raft_consensus.cc:740] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.374854 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7c32c2393a1144e9950b00f61ddba4eb (table default.Table_d89d54d7dc874d7b8d7599cbe59b281a [id=85ee8f90981b4b4f899a45b6c29ff1e0]) successfully deleted
I20251026 02:56:23.375082 2381 consensus_queue.cc:260] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.375589 2372 ts_tablet_manager.cc:1916] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.377960 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_30e98b21e65d443097e2fc01f96f4780"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.378686 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30e98b21e65d443097e2fc01f96f4780 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:23.380383 1647 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 829a6044a2da44ff90acdebca6d5b18d (table default.Table_6546379855b749b889279973db1820cd [id=b8be7adba1cd48ae979de506d892cc3d], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 829a6044a2da44ff90acdebca6d5b18d consensus state has no leader
I20251026 02:56:23.383507 2242 tablet_service.cc:1505] Processing CreateTablet for tablet a821ed8a686c452a975acb96b5c74778 (DEFAULT_TABLE table=default.Table_00dc4c49a19246889353df2157eebf15 [id=347d712aca4e477c89805846800fb824]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.374747 2377 tablet_replica.cc:333] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.384019 2377 raft_consensus.cc:2243] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.384197 2377 raft_consensus.cc:2272] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.384789 2377 ts_tablet_manager.cc:1916] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.386009 2372 ts_tablet_manager.cc:1929] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:23.387290 2376 ts_tablet_manager.cc:1916] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.387554 2378 tablet_bootstrap.cc:492] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:23.387784 2378 ts_tablet_manager.cc:1403] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.175s user 0.008s sys 0.004s
I20251026 02:56:23.388695 2372 log.cc:1199] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/9a04ac20b7624a018c403ef4f64f7a69
I20251026 02:56:23.388985 2378 raft_consensus.cc:359] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:23.389257 2378 raft_consensus.cc:385] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.389302 2372 ts_tablet_manager.cc:1950] T 9a04ac20b7624a018c403ef4f64f7a69 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.389410 2378 raft_consensus.cc:740] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.391027 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 9a04ac20b7624a018c403ef4f64f7a69 (table default.Table_627cb33ef7ce4082b858fa0670d09862 [id=c162ef89d2f14e91a24a6d8c2c37c261]) successfully deleted
I20251026 02:56:23.391201 2377 ts_tablet_manager.cc:1929] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.391330 2377 log.cc:1199] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3ef7247d62ec4adc82fdbed40fc852ea
I20251026 02:56:23.391673 2377 ts_tablet_manager.cc:1950] T 3ef7247d62ec4adc82fdbed40fc852ea P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.389707 2378 consensus_queue.cc:260] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:23.392661 2372 tablet_replica.cc:333] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.394050 2372 raft_consensus.cc:2243] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.394939 2378 ts_tablet_manager.cc:1434] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251026 02:56:23.383626 2381 ts_tablet_manager.cc:1434] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20251026 02:56:23.397058 2378 tablet_bootstrap.cc:492] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:23.397425 2372 raft_consensus.cc:2272] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.398820 2378 tablet_bootstrap.cc:654] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.404975 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a821ed8a686c452a975acb96b5c74778. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.410029 2372 ts_tablet_manager.cc:1916] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.412079 2381 tablet_bootstrap.cc:492] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:23.413187 2377 tablet_replica.cc:333] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.413452 2377 raft_consensus.cc:2243] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.413651 2377 raft_consensus.cc:2272] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.414175 2377 ts_tablet_manager.cc:1916] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.414501 2381 tablet_bootstrap.cc:654] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.416620 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 (DEFAULT_TABLE table=default.Table_25607539f64440cb8ff2c24a0a3a3bdd [id=5dbfb5d912ba433baa46bd52bbf6dbee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.417054 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.421612 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 9156314f98ac4ed9981f6f13fc585043 (DEFAULT_TABLE table=default.Table_4578a4ce81ea42cd878859a8167babff [id=f914c9974ac44d3f84c3d312649fe255]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.421712 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 4011930a277d410f863a1558416c4b0f (DEFAULT_TABLE table=default.Table_8dcca91177f5452e816e2080bceadb93 [id=30cb7a766d854568892ad3a733d34f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.422021 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9156314f98ac4ed9981f6f13fc585043. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.421988 2097 tablet_service.cc:1505] Processing CreateTablet for tablet f8533f1c191042ee9f2c0b99972fe931 (DEFAULT_TABLE table=default.Table_e70cf6d4e0dc4017b484753f4b0250d9 [id=f4fb24ed75a24d81ae8f90619ded2ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.422240 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 22e4a433801a4cb58062d3e76c760b69 (DEFAULT_TABLE table=default.Table_af080a4fb1da461eb6bf487aa8e54990 [id=b4c0611288a949378eed16d32b5170b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.422346 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8533f1c191042ee9f2c0b99972fe931. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.422566 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22e4a433801a4cb58062d3e76c760b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.422834 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 84ad6c72a8704c148411881760848df4 (DEFAULT_TABLE table=default.Table_7814874203fc40af9ef07eeb2e11ebc4 [id=cc22b2daa8fb4eb1bae52b189d66b8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.423193 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ad6c72a8704c148411881760848df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.427155 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4011930a277d410f863a1558416c4b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.430219 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0020cb1f190c49978c5322d387a47f71
I20251026 02:56:23.430672 1654 catalog_manager.cc:6719] ReplaceTablet: tablet 0020cb1f190c49978c5322d387a47f71 deleted and replaced by tablet 86d8e1e21c5f440d8baf76c357d01ad2
I20251026 02:56:23.432072 2405 raft_consensus.cc:493] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:23.431882 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 0020cb1f190c49978c5322d387a47f71 with delete_type TABLET_DATA_DELETED (replaced by tablet 86d8e1e21c5f440d8baf76c357d01ad2) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.432276 2405 raft_consensus.cc:515] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.432938 2405 leader_election.cc:290] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:23.433145 2376 ts_tablet_manager.cc:1929] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.433269 2376 log.cc:1199] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/5ff86fc06e7d46e78fe68cc6f31a30cc
I20251026 02:56:23.433602 2376 ts_tablet_manager.cc:1950] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:23.435506 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b4f977cc4cc4d078f06c4251f8c4fa0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:23.436156 2194 leader_election.cc:343] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:23.437088 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b4f977cc4cc4d078f06c4251f8c4fa0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:23.437670 2195 leader_election.cc:343] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:23.437848 2195 leader_election.cc:304] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:23.438153 2405 raft_consensus.cc:2749] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:23.439962 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a04ac20b7624a018c403ef4f64f7a69 with delete_type TABLET_DATA_DELETED (replaced by tablet 340bf8c78a88413299b61a52f0a578f3) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.445437 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9db88c5b7ec24571b75504849c925cfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.446245 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9db88c5b7ec24571b75504849c925cfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.448539 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 (DEFAULT_TABLE table=default.Table_25607539f64440cb8ff2c24a0a3a3bdd [id=5dbfb5d912ba433baa46bd52bbf6dbee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.449071 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.453127 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 (DEFAULT_TABLE table=default.Table_25607539f64440cb8ff2c24a0a3a3bdd [id=5dbfb5d912ba433baa46bd52bbf6dbee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.453171 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 9156314f98ac4ed9981f6f13fc585043 (DEFAULT_TABLE table=default.Table_4578a4ce81ea42cd878859a8167babff [id=f914c9974ac44d3f84c3d312649fe255]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.453464 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 4011930a277d410f863a1558416c4b0f (DEFAULT_TABLE table=default.Table_8dcca91177f5452e816e2080bceadb93 [id=30cb7a766d854568892ad3a733d34f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.453593 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.453925 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9156314f98ac4ed9981f6f13fc585043. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.454411 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4011930a277d410f863a1558416c4b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.457024 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 9156314f98ac4ed9981f6f13fc585043 (DEFAULT_TABLE table=default.Table_4578a4ce81ea42cd878859a8167babff [id=f914c9974ac44d3f84c3d312649fe255]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.457442 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9156314f98ac4ed9981f6f13fc585043. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.467705 2372 ts_tablet_manager.cc:1929] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.467891 2372 log.cc:1199] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/4eea9b18e5094a22a51942f438a20eb4
I20251026 02:56:23.468329 2372 ts_tablet_manager.cc:1950] T 4eea9b18e5094a22a51942f438a20eb4 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.469599 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 4eea9b18e5094a22a51942f438a20eb4 (table default.Table_71bb5e64488b4f4f92c57f6a1e34fb3c [id=30a69f019af84e09af9f21939f081847]) successfully deleted
I20251026 02:56:23.470091 2372 tablet_replica.cc:333] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.470400 2372 raft_consensus.cc:2243] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.470672 2372 raft_consensus.cc:2272] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.471710 2372 ts_tablet_manager.cc:1916] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.472954 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 22e4a433801a4cb58062d3e76c760b69 (DEFAULT_TABLE table=default.Table_af080a4fb1da461eb6bf487aa8e54990 [id=b4c0611288a949378eed16d32b5170b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.473389 2232 tablet_service.cc:1505] Processing CreateTablet for tablet f8533f1c191042ee9f2c0b99972fe931 (DEFAULT_TABLE table=default.Table_e70cf6d4e0dc4017b484753f4b0250d9 [id=f4fb24ed75a24d81ae8f90619ded2ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.473685 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 84ad6c72a8704c148411881760848df4 (DEFAULT_TABLE table=default.Table_7814874203fc40af9ef07eeb2e11ebc4 [id=cc22b2daa8fb4eb1bae52b189d66b8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.474018 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 4011930a277d410f863a1558416c4b0f (DEFAULT_TABLE table=default.Table_8dcca91177f5452e816e2080bceadb93 [id=30cb7a766d854568892ad3a733d34f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.474086 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ad6c72a8704c148411881760848df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.474537 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 0020cb1f190c49978c5322d387a47f71 with delete_type TABLET_DATA_DELETED (replaced by tablet 86d8e1e21c5f440d8baf76c357d01ad2) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:23.474851 2377 ts_tablet_manager.cc:1929] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.475049 2377 log.cc:1199] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/07280105a2b74ec68e27982414b12902
I20251026 02:56:23.475415 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 22e4a433801a4cb58062d3e76c760b69 (DEFAULT_TABLE table=default.Table_af080a4fb1da461eb6bf487aa8e54990 [id=b4c0611288a949378eed16d32b5170b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.475831 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22e4a433801a4cb58062d3e76c760b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.476060 2377 ts_tablet_manager.cc:1950] T 07280105a2b74ec68e27982414b12902 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.476480 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 84ad6c72a8704c148411881760848df4 (DEFAULT_TABLE table=default.Table_7814874203fc40af9ef07eeb2e11ebc4 [id=cc22b2daa8fb4eb1bae52b189d66b8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.476970 1958 tablet_service.cc:1505] Processing CreateTablet for tablet aa399e6407fd4c4d9871d3d5ccf894ec (DEFAULT_TABLE table=default.Table_65488c1f07e542b5973be5cfd122787c [id=ce3610a9d6814ec783ae7503e58ef3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.477339 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa399e6407fd4c4d9871d3d5ccf894ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.480284 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 0020cb1f190c49978c5322d387a47f71 with delete_type TABLET_DATA_DELETED (replaced by tablet 86d8e1e21c5f440d8baf76c357d01ad2) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.480146 2377 tablet_replica.cc:333] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.480552 1962 tablet_service.cc:1505] Processing CreateTablet for tablet f8533f1c191042ee9f2c0b99972fe931 (DEFAULT_TABLE table=default.Table_e70cf6d4e0dc4017b484753f4b0250d9 [id=f4fb24ed75a24d81ae8f90619ded2ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.480971 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8533f1c191042ee9f2c0b99972fe931. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.480909 2377 raft_consensus.cc:2243] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.481454 2377 raft_consensus.cc:2272] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.484148 2377 ts_tablet_manager.cc:1916] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.486675 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22e4a433801a4cb58062d3e76c760b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.487017 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8533f1c191042ee9f2c0b99972fe931. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.487264 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ad6c72a8704c148411881760848df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.487470 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4011930a277d410f863a1558416c4b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.493600 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 2487bf7d40f34922aa2fd165488342f6
I20251026 02:56:23.494098 1663 catalog_manager.cc:6719] ReplaceTablet: tablet 2487bf7d40f34922aa2fd165488342f6 deleted and replaced by tablet a111848706b743c9beb2fbd21f59d7f3
I20251026 02:56:23.495076 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 2487bf7d40f34922aa2fd165488342f6 with delete_type TABLET_DATA_DELETED (replaced by tablet a111848706b743c9beb2fbd21f59d7f3) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.501983 2377 ts_tablet_manager.cc:1929] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.504242 2372 ts_tablet_manager.cc:1929] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.504482 2372 log.cc:1199] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/79942752cba4429abdc9524f882800da
I20251026 02:56:23.504911 2372 ts_tablet_manager.cc:1950] T 79942752cba4429abdc9524f882800da P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.506027 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 79942752cba4429abdc9524f882800da (table default.Table_a5ff6167974b47798ffe2b3cd583b50b [id=43cb7f538f824ae38059645230cc7e94]) successfully deleted
I20251026 02:56:23.506464 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1972d81a4c494b7c97894d4991fed574"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.506816 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet b108f0d3dae84fab93bff012400c4a2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b108f0d3dae84fab93bff012400c4a2a already in progress: creating tablet
W20251026 02:56:23.507102 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1972d81a4c494b7c97894d4991fed574 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:23.507417 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet b435520cadb344edaa29975b4e19718a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b435520cadb344edaa29975b4e19718a already in progress: creating tablet
I20251026 02:56:23.505584 2405 raft_consensus.cc:493] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:23.507719 2375 tablet_bootstrap.cc:492] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.507859 2375 ts_tablet_manager.cc:1403] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.285s user 0.014s sys 0.057s
I20251026 02:56:23.507814 2405 raft_consensus.cc:515] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.508499 2405 leader_election.cc:290] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:23.508658 2375 raft_consensus.cc:359] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.508846 2375 raft_consensus.cc:385] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.508955 2375 raft_consensus.cc:740] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.509146 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2487bf7d40f34922aa2fd165488342f6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:23.509289 2375 consensus_queue.cc:260] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:23.509675 2195 leader_election.cc:343] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:23.509817 2375 ts_tablet_manager.cc:1434] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:23.510582 2372 tablet_replica.cc:333] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.510818 2372 raft_consensus.cc:2243] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.510962 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2487bf7d40f34922aa2fd165488342f6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:23.511078 2372 raft_consensus.cc:2272] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:23.511833 2194 leader_election.cc:343] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:23.511950 2194 leader_election.cc:304] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:23.512033 2372 ts_tablet_manager.cc:1916] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.512224 2405 raft_consensus.cc:2749] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:23.511471 2377 log.cc:1199] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7b4f977cc4cc4d078f06c4251f8c4fa0
I20251026 02:56:23.512777 2375 tablet_bootstrap.cc:492] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.513162 2377 ts_tablet_manager.cc:1950] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.514796 2375 tablet_bootstrap.cc:654] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.515710 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3d6690e6bf67454398637d9908b9c652
W20251026 02:56:23.519414 1647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.60:33729, user_credentials={real_user=slave}} blocked reactor thread for 44508.4us
I20251026 02:56:23.520170 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 07280105a2b74ec68e27982414b12902 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.520622 2376 tablet_replica.cc:333] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.520833 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 5ff86fc06e7d46e78fe68cc6f31a30cc (table default.Table_d15145ea85264fc9aab94645c7ff3b41 [id=9d7e6bf7d8844942a0221a92da270f3c]) successfully deleted
I20251026 02:56:23.520841 2376 raft_consensus.cc:2243] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.521034 2376 raft_consensus.cc:2272] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:23.521417 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0020cb1f190c49978c5322d387a47f71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0020cb1f190c49978c5322d387a47f71 already in progress: creating tablet
I20251026 02:56:23.521763 2376 ts_tablet_manager.cc:1916] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.522897 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3ef7247d62ec4adc82fdbed40fc852ea (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
I20251026 02:56:23.523378 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 07280105a2b74ec68e27982414b12902 (table default.Table_81ca5851f387446ead27f921baefbe9a [id=6740f80827434cc393266a57238cdbdc]) successfully deleted
I20251026 02:56:23.523947 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 (table default.Table_aa762aa5113a46a98da2f4ded4e23172 [id=2b785b2f8f2d4b0fa3f24b1ea4a66cbf]) successfully deleted
I20251026 02:56:23.527051 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 426ce6fb80874df68210a652f7b7602f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:20 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.527485 2377 tablet_replica.cc:333] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.527735 2377 raft_consensus.cc:2243] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.527979 2377 raft_consensus.cc:2272] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.528375 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 2487bf7d40f34922aa2fd165488342f6 with delete_type TABLET_DATA_DELETED (replaced by tablet a111848706b743c9beb2fbd21f59d7f3) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:23.528350 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3316a969a3654fdf853bca2d3d666aa4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:23.528590 2377 ts_tablet_manager.cc:1916] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:23.528946 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3316a969a3654fdf853bca2d3d666aa4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.529148 2376 ts_tablet_manager.cc:1929] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.529292 2376 log.cc:1199] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/07280105a2b74ec68e27982414b12902
I20251026 02:56:23.529621 2376 ts_tablet_manager.cc:1950] T 07280105a2b74ec68e27982414b12902 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:23.530330 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7dec83bfb9e946db90c10d526af2dfca
I20251026 02:56:23.531149 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dec83bfb9e946db90c10d526af2dfca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.534044 2375 tablet_bootstrap.cc:492] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.534816 2375 ts_tablet_manager.cc:1403] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s
I20251026 02:56:23.535619 2375 raft_consensus.cc:359] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.535898 2377 ts_tablet_manager.cc:1929] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.536070 2377 log.cc:1199] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/5ff86fc06e7d46e78fe68cc6f31a30cc
I20251026 02:56:23.536578 2377 ts_tablet_manager.cc:1950] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.538096 2381 tablet_bootstrap.cc:492] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:23.538300 2381 ts_tablet_manager.cc:1403] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.020s
I20251026 02:56:23.538359 2377 tablet_replica.cc:333] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.538573 2377 raft_consensus.cc:2243] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.538740 2377 raft_consensus.cc:2272] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.538784 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fafb3f2678b846e3bf05a461dde63011"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:23.539176 2381 raft_consensus.cc:359] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.539386 2381 raft_consensus.cc:385] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251026 02:56:23.539438 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fafb3f2678b846e3bf05a461dde63011 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.539469 2381 raft_consensus.cc:740] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.539567 2377 ts_tablet_manager.cc:1916] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.539757 2381 consensus_queue.cc:260] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.543984 2381 ts_tablet_manager.cc:1434] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251026 02:56:23.546241 2377 ts_tablet_manager.cc:1929] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.546388 2377 log.cc:1199] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/e0101098df05462bb26b3c187b259c08
I20251026 02:56:23.546428 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 84ad6c72a8704c148411881760848df4
I20251026 02:56:23.546782 2377 ts_tablet_manager.cc:1950] T e0101098df05462bb26b3c187b259c08 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.547447 2381 tablet_bootstrap.cc:492] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:23.547785 2377 tablet_replica.cc:333] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.547976 2377 raft_consensus.cc:2243] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.548180 2377 raft_consensus.cc:2272] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.548833 2377 ts_tablet_manager.cc:1916] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.549228 2381 tablet_bootstrap.cc:654] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.550400 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d6690e6bf67454398637d9908b9c652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.550597 2375 raft_consensus.cc:385] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.550693 2375 raft_consensus.cc:740] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.550980 2375 consensus_queue.cc:260] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.551690 2375 ts_tablet_manager.cc:1434] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251026 02:56:23.552757 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5d136440d3b64581b112fb1a112950ac" } modify_external_catalogs: true
I20251026 02:56:23.552973 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5d136440d3b64581b112fb1a112950ac" } modify_external_catalogs: true
I20251026 02:56:23.552019 2375 tablet_bootstrap.cc:492] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.556787 2375 tablet_bootstrap.cc:654] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.556876 2378 tablet_bootstrap.cc:492] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:23.557063 2378 ts_tablet_manager.cc:1403] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.160s user 0.003s sys 0.001s
I20251026 02:56:23.557304 2372 ts_tablet_manager.cc:1929] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.557437 2372 log.cc:1199] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/e79223b1086347199470a2526e32bd6f
I20251026 02:56:23.558086 2378 raft_consensus.cc:359] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.558305 2372 ts_tablet_manager.cc:1950] T e79223b1086347199470a2526e32bd6f P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.558346 2378 raft_consensus.cc:385] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.558441 2378 raft_consensus.cc:740] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.558722 2378 consensus_queue.cc:260] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.559521 2372 tablet_replica.cc:333] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.559895 2372 raft_consensus.cc:2243] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.560077 2372 raft_consensus.cc:2272] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.560122 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet e79223b1086347199470a2526e32bd6f (table default.Table_382e6508ce4645699117b0e65595251c [id=214ae71fbaf24f4ab016cb643a89f97f]) successfully deleted
I20251026 02:56:23.560286 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 84ad6c72a8704c148411881760848df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:23.560457 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ea910e98d17340efb816a8aa90f0ba7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea910e98d17340efb816a8aa90f0ba7e already in progress: creating tablet
I20251026 02:56:23.562189 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d6690e6bf67454398637d9908b9c652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.562493 2372 ts_tablet_manager.cc:1916] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.566143 2378 ts_tablet_manager.cc:1434] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20251026 02:56:23.568774 2090 tablet_service.cc:1505] Processing CreateTablet for tablet aa399e6407fd4c4d9871d3d5ccf894ec (DEFAULT_TABLE table=default.Table_65488c1f07e542b5973be5cfd122787c [id=ce3610a9d6814ec783ae7503e58ef3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.569186 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa399e6407fd4c4d9871d3d5ccf894ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.569986 2378 tablet_bootstrap.cc:492] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:23.571058 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 47bfff40c181432896d6898ea50cd025 (DEFAULT_TABLE table=default.Table_d9f69f33702146e281d6d9bfa43e968b [id=6eab78e6aeb344d8bef3c455d8df684d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.571492 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47bfff40c181432896d6898ea50cd025. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.571924 2378 tablet_bootstrap.cc:654] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.572368 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 2487bf7d40f34922aa2fd165488342f6 with delete_type TABLET_DATA_DELETED (replaced by tablet a111848706b743c9beb2fbd21f59d7f3) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.573238 2372 ts_tablet_manager.cc:1929] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.573418 2372 log.cc:1199] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7c32c2393a1144e9950b00f61ddba4eb
I20251026 02:56:23.573836 2372 ts_tablet_manager.cc:1950] T 7c32c2393a1144e9950b00f61ddba4eb P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.575043 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7c32c2393a1144e9950b00f61ddba4eb (table default.Table_d89d54d7dc874d7b8d7599cbe59b281a [id=85ee8f90981b4b4f899a45b6c29ff1e0]) successfully deleted
W20251026 02:56:23.575944 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0020cb1f190c49978c5322d387a47f71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0020cb1f190c49978c5322d387a47f71 already in progress: creating tablet
W20251026 02:56:23.576678 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9a04ac20b7624a018c403ef4f64f7a69 because the tablet was not found. No further retry: Not found: Tablet not found: 9a04ac20b7624a018c403ef4f64f7a69
W20251026 02:56:23.577371 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 2487bf7d40f34922aa2fd165488342f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2487bf7d40f34922aa2fd165488342f6 already in progress: creating tablet
W20251026 02:56:23.577598 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1a2522801364693be0bda6fa45a697e has no consensus state
I20251026 02:56:23.577730 2372 tablet_replica.cc:333] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.577956 2372 raft_consensus.cc:2243] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.578369 2372 raft_consensus.cc:2272] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.579757 2372 ts_tablet_manager.cc:1916] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:23.580495 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a70d70abc3a34dac8cf9a747903fa78c has no consensus state
I20251026 02:56:23.581535 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 5ff86fc06e7d46e78fe68cc6f31a30cc (table default.Table_d15145ea85264fc9aab94645c7ff3b41 [id=9d7e6bf7d8844942a0221a92da270f3c]) successfully deleted
W20251026 02:56:23.581764 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 1c1596dd4394419a8bb6ea6a84be9fea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c1596dd4394419a8bb6ea6a84be9fea already in progress: creating tablet
W20251026 02:56:23.581979 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 461412f3ad1e4186b1302cdf5a624ed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 461412f3ad1e4186b1302cdf5a624ed4 already in progress: creating tablet
I20251026 02:56:23.582190 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet e0101098df05462bb26b3c187b259c08 (table default.Table_bcf0aac1b41747e9a94a97b8430fa081 [id=df8666bd175f43ceb47649f67a3d865e]) successfully deleted
I20251026 02:56:23.582363 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 07280105a2b74ec68e27982414b12902 (table default.Table_81ca5851f387446ead27f921baefbe9a [id=6740f80827434cc393266a57238cdbdc]) successfully deleted
I20251026 02:56:23.582540 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dec83bfb9e946db90c10d526af2dfca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:23.582715 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3d6690e6bf67454398637d9908b9c652 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d6690e6bf67454398637d9908b9c652 already in progress: creating tablet
I20251026 02:56:23.582774 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9709925e5dd846f78b29265c3a7635dc" } modify_external_catalogs: true
W20251026 02:56:23.582922 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 2487bf7d40f34922aa2fd165488342f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2487bf7d40f34922aa2fd165488342f6 already in progress: creating tablet
I20251026 02:56:23.582907 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d6690e6bf67454398637d9908b9c652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:23.582983 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9709925e5dd846f78b29265c3a7635dc" } modify_external_catalogs: true
I20251026 02:56:23.582918 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 84ad6c72a8704c148411881760848df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:23.583269 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fe6cbe999c54b299d729e97f0f938e7 with delete_type TABLET_DATA_DELETED (replaced by tablet ad55040f0a2b4cfa894d1bd058d4d69f) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.583297 2233 tablet_service.cc:1505] Processing CreateTablet for tablet aa399e6407fd4c4d9871d3d5ccf894ec (DEFAULT_TABLE table=default.Table_65488c1f07e542b5973be5cfd122787c [id=ce3610a9d6814ec783ae7503e58ef3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.583675 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa399e6407fd4c4d9871d3d5ccf894ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.584343 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ee0a408253e4f2b942c510ae64c703e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:23.584621 2381 tablet_bootstrap.cc:492] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:23.584756 2381 ts_tablet_manager.cc:1403] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20251026 02:56:23.585304 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 47bfff40c181432896d6898ea50cd025 (DEFAULT_TABLE table=default.Table_d9f69f33702146e281d6d9bfa43e968b [id=6eab78e6aeb344d8bef3c455d8df684d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.585685 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47bfff40c181432896d6898ea50cd025. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.585641 2381 raft_consensus.cc:359] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.585827 2381 raft_consensus.cc:385] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.585925 2381 raft_consensus.cc:740] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.588354 2377 ts_tablet_manager.cc:1929] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.588372 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 84ad6c72a8704c148411881760848df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.589560 2372 ts_tablet_manager.cc:1929] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.589167 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1e459a892e604d56aeb212131b5a624f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:23.589697 2372 log.cc:1199] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/426ce6fb80874df68210a652f7b7602f
I20251026 02:56:23.590065 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef7247d62ec4adc82fdbed40fc852ea with delete_type TABLET_DATA_DELETED (replaced by tablet 0c647782049243d6bcd595f1a10d89c6) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.590288 2372 ts_tablet_manager.cc:1950] T 426ce6fb80874df68210a652f7b7602f P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.590341 2376 tablet_replica.cc:333] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.590502 2376 raft_consensus.cc:2243] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:23.590523 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e459a892e604d56aeb212131b5a624f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.590700 2376 raft_consensus.cc:2272] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.591323 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 426ce6fb80874df68210a652f7b7602f (table default.Table_ff61ac36382f428598c86ecb81001e88 [id=6071000b0280484b8be3da135c7f7b74]) successfully deleted
I20251026 02:56:23.591429 2376 ts_tablet_manager.cc:1916] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:23.591948 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7dec83bfb9e946db90c10d526af2dfca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dec83bfb9e946db90c10d526af2dfca already in progress: creating tablet
W20251026 02:56:23.592562 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3d6690e6bf67454398637d9908b9c652 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d6690e6bf67454398637d9908b9c652 already in progress: creating tablet
W20251026 02:56:23.593189 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 84ad6c72a8704c148411881760848df4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84ad6c72a8704c148411881760848df4 already in progress: creating tablet
I20251026 02:56:23.593451 2372 tablet_replica.cc:333] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.593632 2372 raft_consensus.cc:2243] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.593777 2372 raft_consensus.cc:2272] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.594527 2372 ts_tablet_manager.cc:1916] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.592556 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 47bfff40c181432896d6898ea50cd025 (DEFAULT_TABLE table=default.Table_d9f69f33702146e281d6d9bfa43e968b [id=6eab78e6aeb344d8bef3c455d8df684d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.595003 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47bfff40c181432896d6898ea50cd025. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.598088 2376 ts_tablet_manager.cc:1929] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.598467 2376 log.cc:1199] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3ef7247d62ec4adc82fdbed40fc852ea
I20251026 02:56:23.598927 2376 ts_tablet_manager.cc:1950] T 3ef7247d62ec4adc82fdbed40fc852ea P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:23.586403 2381 consensus_queue.cc:260] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.600669 2381 ts_tablet_manager.cc:1434] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
W20251026 02:56:23.600625 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 84ad6c72a8704c148411881760848df4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84ad6c72a8704c148411881760848df4 already in progress: creating tablet
I20251026 02:56:23.600978 2381 tablet_bootstrap.cc:492] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:23.601151 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3ef7247d62ec4adc82fdbed40fc852ea (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
I20251026 02:56:23.602774 2381 tablet_bootstrap.cc:654] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.603399 2372 ts_tablet_manager.cc:1929] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.603555 2372 log.cc:1199] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7fe6cbe999c54b299d729e97f0f938e7
I20251026 02:56:23.603752 2377 log.cc:1199] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/0e89c69e3b58458aad10beaeb5128d60
I20251026 02:56:23.603919 2372 ts_tablet_manager.cc:1950] T 7fe6cbe999c54b299d729e97f0f938e7 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.604139 2377 ts_tablet_manager.cc:1950] T 0e89c69e3b58458aad10beaeb5128d60 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.605209 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 0e89c69e3b58458aad10beaeb5128d60 (table default.Table_00dc4c49a19246889353df2157eebf15 [id=347d712aca4e477c89805846800fb824]) successfully deleted
I20251026 02:56:23.605257 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7fe6cbe999c54b299d729e97f0f938e7 (table default.Table_0da51595453549b3b7e077f7b7ff3ec7 [id=53cd517e68e64597ad51d85d16f0399d]) successfully deleted
I20251026 02:56:23.606659 2377 tablet_replica.cc:333] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539: stopping tablet replica
W20251026 02:56:23.606740 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f60bcc5e4cb4fd19d2a5713f69814cd already in progress: creating tablet
I20251026 02:56:23.606873 2377 raft_consensus.cc:2243] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:23.607014 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ac19c5a847454a23ab16831198ff2a20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac19c5a847454a23ab16831198ff2a20 already in progress: creating tablet
I20251026 02:56:23.607046 2377 raft_consensus.cc:2272] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.607579 2377 ts_tablet_manager.cc:1916] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.614327 2375 tablet_bootstrap.cc:492] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.614487 2375 ts_tablet_manager.cc:1403] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.000s
I20251026 02:56:23.615315 2375 raft_consensus.cc:359] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.615525 2375 raft_consensus.cc:385] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.615614 2375 raft_consensus.cc:740] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.615852 2375 consensus_queue.cc:260] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.619012 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dec83bfb9e946db90c10d526af2dfca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:23.620815 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7dec83bfb9e946db90c10d526af2dfca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dec83bfb9e946db90c10d526af2dfca already in progress: creating tablet
I20251026 02:56:23.623793 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:23.629287 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 0020cb1f190c49978c5322d387a47f71 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:23.631201 1659 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 17137a72b8f7404cab6cb49c56f9bae6
I20251026 02:56:23.631445 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:23.631814 2375 ts_tablet_manager.cc:1434] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251026 02:56:23.633456 2375 tablet_bootstrap.cc:492] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.635161 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_953a107579574f9088605b5e07dd917e" } new_table_name: "default.Table_95868643edf84f8a8a325f3cb56aa732" modify_external_catalogs: true
I20251026 02:56:23.636411 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef7247d62ec4adc82fdbed40fc852ea with delete_type TABLET_DATA_DELETED (replaced by tablet 0c647782049243d6bcd595f1a10d89c6) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.636092 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e2a5e05b70f841cfb0065041b39e89cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.637204 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2a5e05b70f841cfb0065041b39e89cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.637909 2372 tablet_replica.cc:333] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.638093 2372 raft_consensus.cc:2243] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.638285 2372 raft_consensus.cc:2272] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.638358 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_627cb33ef7ce4082b858fa0670d09862" } new_table_name: "default.Table_42e1adcd9ad84e029d7706cc7fd847a1" modify_external_catalogs: true
I20251026 02:56:23.636364 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_de29df31fe874815a8f1b1d25f4ba210" } new_table_name: "default.Table_08b776b4561f401880b0c7ba0a9b6fa2" modify_external_catalogs: true
I20251026 02:56:23.639662 2372 ts_tablet_manager.cc:1916] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.642956 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7b6687267a1a4b2da7e02949e7b4c081"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.643632 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b6687267a1a4b2da7e02949e7b4c081 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.650372 2381 tablet_bootstrap.cc:492] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:23.651815 2378 tablet_bootstrap.cc:492] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:23.652040 2378 ts_tablet_manager.cc:1403] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.000s
I20251026 02:56:23.655040 2375 tablet_bootstrap.cc:654] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.655267 2381 ts_tablet_manager.cc:1403] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.005s
I20251026 02:56:23.656108 2381 raft_consensus.cc:359] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.656981 2378 raft_consensus.cc:359] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.657270 2378 raft_consensus.cc:385] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.657369 2378 raft_consensus.cc:740] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.657693 2378 consensus_queue.cc:260] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.655889 2372 ts_tablet_manager.cc:1929] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.658057 2372 log.cc:1199] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3ef7247d62ec4adc82fdbed40fc852ea
I20251026 02:56:23.658128 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef7247d62ec4adc82fdbed40fc852ea with delete_type TABLET_DATA_DELETED (replaced by tablet 0c647782049243d6bcd595f1a10d89c6) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.658337 2381 raft_consensus.cc:385] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251026 02:56:23.658879 1647 catalog_manager.cc:4952] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3ef7247d62ec4adc82fdbed40fc852ea because the tablet was not found. No further retry: Not found: Tablet not found: 3ef7247d62ec4adc82fdbed40fc852ea
I20251026 02:56:23.659346 2372 ts_tablet_manager.cc:1950] T 3ef7247d62ec4adc82fdbed40fc852ea P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.659466 2381 raft_consensus.cc:740] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.663439 2381 consensus_queue.cc:260] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.664865 2381 ts_tablet_manager.cc:1434] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251026 02:56:23.665848 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3ef7247d62ec4adc82fdbed40fc852ea (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
I20251026 02:56:23.666046 2381 tablet_bootstrap.cc:492] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:23.660771 2377 ts_tablet_manager.cc:1929] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.666486 2377 log.cc:1199] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/c0f83d631faa468f828e54e97987a00b
I20251026 02:56:23.666296 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet e0101098df05462bb26b3c187b259c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.666945 2378 ts_tablet_manager.cc:1434] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
W20251026 02:56:23.667366 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet e0101098df05462bb26b3c187b259c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0101098df05462bb26b3c187b259c08 already in progress: creating tablet
I20251026 02:56:23.668983 2377 ts_tablet_manager.cc:1950] T c0f83d631faa468f828e54e97987a00b P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.671573 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet e0101098df05462bb26b3c187b259c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.671676 2378 tablet_bootstrap.cc:492] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:23.671824 2376 tablet_replica.cc:333] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.672020 2376 raft_consensus.cc:2243] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.672236 2376 raft_consensus.cc:2272] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.672883 2376 ts_tablet_manager.cc:1916] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.676889 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet c0f83d631faa468f828e54e97987a00b (table default.Table_4578a4ce81ea42cd878859a8167babff [id=f914c9974ac44d3f84c3d312649fe255]) successfully deleted
I20251026 02:56:23.676991 2377 tablet_replica.cc:333] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.677201 2377 raft_consensus.cc:2243] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.677412 2377 raft_consensus.cc:2272] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:23.677536 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 20dec74440cf4bb9949c462ec51e9889 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20dec74440cf4bb9949c462ec51e9889 already in progress: creating tablet
I20251026 02:56:23.677559 2378 tablet_bootstrap.cc:654] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.677989 2377 ts_tablet_manager.cc:1916] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.680122 2376 ts_tablet_manager.cc:1929] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.668390 2381 tablet_bootstrap.cc:654] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.680541 2376 log.cc:1199] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/e0101098df05462bb26b3c187b259c08
I20251026 02:56:23.680979 2376 ts_tablet_manager.cc:1950] T e0101098df05462bb26b3c187b259c08 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:23.682754 2375 tablet_bootstrap.cc:492] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.682930 2375 ts_tablet_manager.cc:1403] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.002s
I20251026 02:56:23.683692 2375 raft_consensus.cc:359] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.683878 2375 raft_consensus.cc:385] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.683950 2375 raft_consensus.cc:740] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.684259 2375 consensus_queue.cc:260] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.685670 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet e0101098df05462bb26b3c187b259c08 (table default.Table_bcf0aac1b41747e9a94a97b8430fa081 [id=df8666bd175f43ceb47649f67a3d865e]) successfully deleted
I20251026 02:56:23.687400 2375 ts_tablet_manager.cc:1434] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251026 02:56:23.689409 2375 tablet_bootstrap.cc:492] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.692759 2375 tablet_bootstrap.cc:654] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.696399 1661 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 9c19b416ecf34d46847ebcd08987bd71
I20251026 02:56:23.697829 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 3d6690e6bf67454398637d9908b9c652 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:23.701227 1632 master_runner.cc:186] Error getting master registration for 127.1.150.60:33729: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111),
I20251026 02:56:23.701431 1632 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 02:56:23.701841 2375 tablet_bootstrap.cc:492] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.696271 1700 master_runner.cc:186] Error getting master registration for 127.1.150.60:33729: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111),
I20251026 02:56:23.702920 1700 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 02:56:23.704958 2377 ts_tablet_manager.cc:1929] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.705114 2377 log.cc:1199] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/565255802f4f42ba8320a3ec80d2db8f
I20251026 02:56:23.705461 2377 ts_tablet_manager.cc:1950] T 565255802f4f42ba8320a3ec80d2db8f P 696476265495492790739ec126a08539: Deleting consensus metadata
W20251026 02:56:23.705919 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 340bf8c78a88413299b61a52f0a578f3 consensus state has no leader
I20251026 02:56:23.706610 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 565255802f4f42ba8320a3ec80d2db8f (table default.Table_cf4652bce4e447caa066bfebc7b43e50 [id=1f46f0e30c4045419969ccc51df359f1]) successfully deleted
I20251026 02:56:23.706883 2405 raft_consensus.cc:493] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:23.707106 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 865fbd674db34f668fb5a2f5836a9b43 has no consensus state
I20251026 02:56:23.707051 2405 raft_consensus.cc:515] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.707669 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet c1b6257a3c8c4e1a8144ea45ff2a2a96
I20251026 02:56:23.707691 2405 leader_election.cc:290] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:23.708086 1662 catalog_manager.cc:6719] ReplaceTablet: tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 deleted and replaced by tablet 94886ad6d6bb45f39261725169774ac9
I20251026 02:56:23.708577 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ee0a408253e4f2b942c510ae64c703e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:23.708681 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5
I20251026 02:56:23.708707 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 with delete_type TABLET_DATA_DELETED (replaced by tablet 94886ad6d6bb45f39261725169774ac9) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:23.709306 1659 catalog_manager.cc:6719] ReplaceTablet: tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 deleted and replaced by tablet 1e56b259a314422aba35eb129021bbeb
I20251026 02:56:23.709649 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e56b259a314422aba35eb129021bbeb) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:23.709697 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 903e65fa8c214dc9a50cb97cab101d22 consensus state has no leader
I20251026 02:56:23.709744 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e56b259a314422aba35eb129021bbeb) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.710038 2377 tablet_replica.cc:333] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539: stopping tablet replica
W20251026 02:56:23.710131 2194 leader_election.cc:343] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:23.710273 2377 raft_consensus.cc:2243] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.710530 2377 raft_consensus.cc:2272] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.710662 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 with delete_type TABLET_DATA_DELETED (replaced by tablet 94886ad6d6bb45f39261725169774ac9) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.711083 2377 ts_tablet_manager.cc:1916] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.717790 2377 ts_tablet_manager.cc:1929] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.717939 2377 log.cc:1199] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/b108f0d3dae84fab93bff012400c4a2a
I20251026 02:56:23.718302 2377 ts_tablet_manager.cc:1950] T b108f0d3dae84fab93bff012400c4a2a P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.719280 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet b108f0d3dae84fab93bff012400c4a2a (table default.Table_fa379fbb05ce4e46a08b7fc1bae3183a [id=2f5d9104d21c49e0985a01ffb4b7a978]) successfully deleted
I20251026 02:56:23.719635 2377 tablet_replica.cc:333] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.719812 2377 raft_consensus.cc:2243] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.719990 2377 raft_consensus.cc:2272] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.720827 2377 ts_tablet_manager.cc:1916] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:23.725251 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 already in progress: creating tablet
W20251026 02:56:23.726154 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 already in progress: creating tablet
I20251026 02:56:23.727330 2377 ts_tablet_manager.cc:1929] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.727480 2377 log.cc:1199] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/b435520cadb344edaa29975b4e19718a
I20251026 02:56:23.727864 2377 ts_tablet_manager.cc:1950] T b435520cadb344edaa29975b4e19718a P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.728930 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet b435520cadb344edaa29975b4e19718a (table default.Table_66dd770512e847f4a14ee04e5afa3a6c [id=72fa811ef7224d42a83e602c3707f028]) successfully deleted
W20251026 02:56:23.729552 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ea910e98d17340efb816a8aa90f0ba7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea910e98d17340efb816a8aa90f0ba7e already in progress: creating tablet
W20251026 02:56:23.730084 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0020cb1f190c49978c5322d387a47f71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0020cb1f190c49978c5322d387a47f71 already in progress: creating tablet
I20251026 02:56:23.730347 2377 tablet_replica.cc:333] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.730558 2377 raft_consensus.cc:2243] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.730835 2377 raft_consensus.cc:2272] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.731840 2381 tablet_bootstrap.cc:492] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:23.702013 2375 ts_tablet_manager.cc:1403] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20251026 02:56:23.709045 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ee0a408253e4f2b942c510ae64c703e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:23.732016 2381 ts_tablet_manager.cc:1403] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.005s
I20251026 02:56:23.711483 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 with delete_type TABLET_DATA_DELETED (replaced by tablet 94886ad6d6bb45f39261725169774ac9) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.712867 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e56b259a314422aba35eb129021bbeb) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.732820 2375 raft_consensus.cc:359] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:23.732828 2381 raft_consensus.cc:359] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:23.733018 2381 raft_consensus.cc:385] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.702590 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ff86fc06e7d46e78fe68cc6f31a30cc with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.733104 2381 raft_consensus.cc:740] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.733322 2381 consensus_queue.cc:260] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:23.733685 2195 leader_election.cc:343] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:23.733867 2381 ts_tablet_manager.cc:1434] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:23.733824 2195 leader_election.cc:304] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:23.734182 2381 tablet_bootstrap.cc:492] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:23.734584 2440 raft_consensus.cc:2749] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:23.735504 2372 tablet_replica.cc:333] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.735687 2372 raft_consensus.cc:2243] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.735852 2372 raft_consensus.cc:2272] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.736613 2372 ts_tablet_manager.cc:1916] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:23.736627 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 already in progress: creating tablet
I20251026 02:56:23.736917 2381 tablet_bootstrap.cc:654] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:23.736928 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 already in progress: creating tablet
I20251026 02:56:23.733000 2375 raft_consensus.cc:385] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.740476 2375 raft_consensus.cc:740] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.740500 2377 ts_tablet_manager.cc:1916] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.740844 2375 consensus_queue.cc:260] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:23.742892 2375 ts_tablet_manager.cc:1434] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251026 02:56:23.743906 2375 tablet_bootstrap.cc:492] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.745069 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 07280105a2b74ec68e27982414b12902 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.745827 2375 tablet_bootstrap.cc:654] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.752940 2372 ts_tablet_manager.cc:1929] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.753262 2378 tablet_bootstrap.cc:492] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:23.753428 2378 ts_tablet_manager.cc:1403] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.002s
I20251026 02:56:23.754259 2372 log.cc:1199] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/5ff86fc06e7d46e78fe68cc6f31a30cc
I20251026 02:56:23.754285 2378 raft_consensus.cc:359] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.754462 2377 ts_tablet_manager.cc:1929] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.754508 2378 raft_consensus.cc:385] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.754611 2377 log.cc:1199] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/2487bf7d40f34922aa2fd165488342f6
I20251026 02:56:23.754603 2378 raft_consensus.cc:740] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.754858 2378 consensus_queue.cc:260] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.755164 2372 ts_tablet_manager.cc:1950] T 5ff86fc06e7d46e78fe68cc6f31a30cc P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.755399 2378 ts_tablet_manager.cc:1434] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:23.756263 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 5ff86fc06e7d46e78fe68cc6f31a30cc (table default.Table_d15145ea85264fc9aab94645c7ff3b41 [id=9d7e6bf7d8844942a0221a92da270f3c]) successfully deleted
I20251026 02:56:23.758155 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 358e805370ef410e9d65ee92a8d04f34 (DEFAULT_TABLE table=default.Table_cba8c65ea6a14b09a43a61d89e0ecee4 [id=e86d9fc892a8409590c555e25b27a12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.758584 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358e805370ef410e9d65ee92a8d04f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.759334 2372 tablet_replica.cc:333] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:23.759493 2377 ts_tablet_manager.cc:1950] T 2487bf7d40f34922aa2fd165488342f6 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.759532 2372 raft_consensus.cc:2243] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.759747 2372 raft_consensus.cc:2272] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.760694 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 829a6044a2da44ff90acdebca6d5b18d (table default.Table_6546379855b749b889279973db1820cd [id=b8be7adba1cd48ae979de506d892cc3d]). Expected version 2 got 0
I20251026 02:56:23.762527 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 358e805370ef410e9d65ee92a8d04f34 (DEFAULT_TABLE table=default.Table_cba8c65ea6a14b09a43a61d89e0ecee4 [id=e86d9fc892a8409590c555e25b27a12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.762988 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358e805370ef410e9d65ee92a8d04f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.763110 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 358e805370ef410e9d65ee92a8d04f34 (DEFAULT_TABLE table=default.Table_cba8c65ea6a14b09a43a61d89e0ecee4 [id=e86d9fc892a8409590c555e25b27a12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.763366 2378 tablet_bootstrap.cc:492] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:23.763348 2232 tablet_service.cc:1505] Processing CreateTablet for tablet ca4611c0a494425ca50d599b310872c8 (DEFAULT_TABLE table=default.Table_4c63b6bed4b1437f9a11cba9c5d09d15 [id=2dcd0d98a1354635a3673182e6fad58a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.763494 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358e805370ef410e9d65ee92a8d04f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.763679 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca4611c0a494425ca50d599b310872c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.765609 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 829a6044a2da44ff90acdebca6d5b18d (table default.Table_6546379855b749b889279973db1820cd [id=b8be7adba1cd48ae979de506d892cc3d]). Expected version 2 got 0
I20251026 02:56:23.765717 2378 tablet_bootstrap.cc:654] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.768558 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 5226d4516ce24ecda51dab35c86da132 (DEFAULT_TABLE table=default.Table_5cda8845fc8f42e9a04ed877e65b2be3 [id=c8435cef755d4de69c33f599df1c0f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.768613 1958 tablet_service.cc:1505] Processing CreateTablet for tablet ca4611c0a494425ca50d599b310872c8 (DEFAULT_TABLE table=default.Table_4c63b6bed4b1437f9a11cba9c5d09d15 [id=2dcd0d98a1354635a3673182e6fad58a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.768939 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5226d4516ce24ecda51dab35c86da132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.769040 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca4611c0a494425ca50d599b310872c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.772096 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 865fbd674db34f668fb5a2f5836a9b43 (DEFAULT_TABLE table=default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.772488 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 865fbd674db34f668fb5a2f5836a9b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.772497 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 5226d4516ce24ecda51dab35c86da132 (DEFAULT_TABLE table=default.Table_5cda8845fc8f42e9a04ed877e65b2be3 [id=c8435cef755d4de69c33f599df1c0f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.772862 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5226d4516ce24ecda51dab35c86da132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.772835 2239 tablet_service.cc:1505] Processing CreateTablet for tablet dbfe80d388644d0488cb2431a334c908 (DEFAULT_TABLE table=default.Table_ce7dea3ace314a5fa3ffeb9f16512725 [id=5964dceedb1c4d69b39051f9fe86b9dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.773267 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbfe80d388644d0488cb2431a334c908. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.776517 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 865fbd674db34f668fb5a2f5836a9b43 (DEFAULT_TABLE table=default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.776963 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 865fbd674db34f668fb5a2f5836a9b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.777122 2237 tablet_service.cc:1505] Processing CreateTablet for tablet a70d70abc3a34dac8cf9a747903fa78c (DEFAULT_TABLE table=default.Table_54299d87a8e24d0484cc7f08d2b2e1f4 [id=a84628b0b09f43c69e86344d40641b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.777592 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70d70abc3a34dac8cf9a747903fa78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.782505 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_54299d87a8e24d0484cc7f08d2b2e1f4" } new_table_name: "default.Table_c4bf189953554cb8854613da705a2b42" modify_external_catalogs: true
I20251026 02:56:23.782779 2232 tablet_service.cc:1505] Processing CreateTablet for tablet a111848706b743c9beb2fbd21f59d7f3 (DEFAULT_TABLE table=default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.783206 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a111848706b743c9beb2fbd21f59d7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.783897 2095 tablet_service.cc:1505] Processing CreateTablet for tablet ca4611c0a494425ca50d599b310872c8 (DEFAULT_TABLE table=default.Table_4c63b6bed4b1437f9a11cba9c5d09d15 [id=2dcd0d98a1354635a3673182e6fad58a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.784014 2099 tablet_service.cc:1505] Processing CreateTablet for tablet a70d70abc3a34dac8cf9a747903fa78c (DEFAULT_TABLE table=default.Table_54299d87a8e24d0484cc7f08d2b2e1f4 [id=a84628b0b09f43c69e86344d40641b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.784363 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca4611c0a494425ca50d599b310872c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.784617 2098 tablet_service.cc:1505] Processing CreateTablet for tablet dbfe80d388644d0488cb2431a334c908 (DEFAULT_TABLE table=default.Table_ce7dea3ace314a5fa3ffeb9f16512725 [id=5964dceedb1c4d69b39051f9fe86b9dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.784966 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbfe80d388644d0488cb2431a334c908. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.788525 2092 tablet_service.cc:1505] Processing CreateTablet for tablet a111848706b743c9beb2fbd21f59d7f3 (DEFAULT_TABLE table=default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.788786 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 865fbd674db34f668fb5a2f5836a9b43 (DEFAULT_TABLE table=default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.788960 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a111848706b743c9beb2fbd21f59d7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.789206 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 865fbd674db34f668fb5a2f5836a9b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.788527 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 5226d4516ce24ecda51dab35c86da132 (DEFAULT_TABLE table=default.Table_5cda8845fc8f42e9a04ed877e65b2be3 [id=c8435cef755d4de69c33f599df1c0f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.790117 2239 tablet_service.cc:1505] Processing CreateTablet for tablet b1a2522801364693be0bda6fa45a697e (DEFAULT_TABLE table=default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.790795 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5226d4516ce24ecda51dab35c86da132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.790925 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1a2522801364693be0bda6fa45a697e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.782559 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c46d7af7b60a48188efbea88a1b7a72b" } modify_external_catalogs: true
I20251026 02:56:23.794476 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c46d7af7b60a48188efbea88a1b7a72b" } modify_external_catalogs: true
I20251026 02:56:23.795161 1951 tablet_service.cc:1505] Processing CreateTablet for tablet a70d70abc3a34dac8cf9a747903fa78c (DEFAULT_TABLE table=default.Table_54299d87a8e24d0484cc7f08d2b2e1f4 [id=a84628b0b09f43c69e86344d40641b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.795586 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70d70abc3a34dac8cf9a747903fa78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.795718 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet c0f83d631faa468f828e54e97987a00b with delete_type TABLET_DATA_DELETED (replaced by tablet 9156314f98ac4ed9981f6f13fc585043) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:23.795998 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70d70abc3a34dac8cf9a747903fa78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.796375 2376 tablet_replica.cc:333] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:23.796567 2376 raft_consensus.cc:2243] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.796779 2376 raft_consensus.cc:2272] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.800540 1955 tablet_service.cc:1505] Processing CreateTablet for tablet dbfe80d388644d0488cb2431a334c908 (DEFAULT_TABLE table=default.Table_ce7dea3ace314a5fa3ffeb9f16512725 [id=5964dceedb1c4d69b39051f9fe86b9dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.800956 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbfe80d388644d0488cb2431a334c908. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.804543 1961 tablet_service.cc:1505] Processing CreateTablet for tablet a111848706b743c9beb2fbd21f59d7f3 (DEFAULT_TABLE table=default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.804955 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a111848706b743c9beb2fbd21f59d7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.806685 2377 tablet_replica.cc:333] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:23.806867 2377 raft_consensus.cc:2243] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:23.807052 2377 raft_consensus.cc:2272] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:23.808697 2377 ts_tablet_manager.cc:1916] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.808634 1952 tablet_service.cc:1505] Processing CreateTablet for tablet b1a2522801364693be0bda6fa45a697e (DEFAULT_TABLE table=default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.809176 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1a2522801364693be0bda6fa45a697e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.812649 1948 tablet_service.cc:1505] Processing CreateTablet for tablet e62064fbdffb4954a3a1227efd6327ee (DEFAULT_TABLE table=default.Table_acad6a9e93e24617aff27d46e9c5c099 [id=fb1dbbc49ad94d85925be7fc850041e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.813167 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e62064fbdffb4954a3a1227efd6327ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.817890 2097 tablet_service.cc:1505] Processing CreateTablet for tablet b1a2522801364693be0bda6fa45a697e (DEFAULT_TABLE table=default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.818048 2091 tablet_service.cc:1505] Processing CreateTablet for tablet e62064fbdffb4954a3a1227efd6327ee (DEFAULT_TABLE table=default.Table_acad6a9e93e24617aff27d46e9c5c099 [id=fb1dbbc49ad94d85925be7fc850041e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.818521 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1a2522801364693be0bda6fa45a697e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.819444 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 2487bf7d40f34922aa2fd165488342f6 (table default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]) successfully deleted
W20251026 02:56:23.819772 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7dec83bfb9e946db90c10d526af2dfca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dec83bfb9e946db90c10d526af2dfca already in progress: creating tablet
W20251026 02:56:23.820055 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3d6690e6bf67454398637d9908b9c652 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d6690e6bf67454398637d9908b9c652 already in progress: creating tablet
I20251026 02:56:23.820664 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 1370b9ad917344c093e5da3d0e9b1cf0 (DEFAULT_TABLE table=default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.820875 2232 tablet_service.cc:1505] Processing CreateTablet for tablet e62064fbdffb4954a3a1227efd6327ee (DEFAULT_TABLE table=default.Table_acad6a9e93e24617aff27d46e9c5c099 [id=fb1dbbc49ad94d85925be7fc850041e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.820961 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 1370b9ad917344c093e5da3d0e9b1cf0 (DEFAULT_TABLE table=default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.821188 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 86d8e1e21c5f440d8baf76c357d01ad2 (DEFAULT_TABLE table=default.Table_689f802c9e4b457eb71fb78898fe14b0 [id=05242557de9a4a19948b8591ce73ab28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.821305 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e62064fbdffb4954a3a1227efd6327ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.821516 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 86d8e1e21c5f440d8baf76c357d01ad2 (DEFAULT_TABLE table=default.Table_689f802c9e4b457eb71fb78898fe14b0 [id=05242557de9a4a19948b8591ce73ab28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.821537 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 1370b9ad917344c093e5da3d0e9b1cf0 (DEFAULT_TABLE table=default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.821867 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86d8e1e21c5f440d8baf76c357d01ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.821918 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1370b9ad917344c093e5da3d0e9b1cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.821933 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1370b9ad917344c093e5da3d0e9b1cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.824381 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1370b9ad917344c093e5da3d0e9b1cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.824610 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 86d8e1e21c5f440d8baf76c357d01ad2 (DEFAULT_TABLE table=default.Table_689f802c9e4b457eb71fb78898fe14b0 [id=05242557de9a4a19948b8591ce73ab28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:23.825083 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86d8e1e21c5f440d8baf76c357d01ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.825778 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86d8e1e21c5f440d8baf76c357d01ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.829501 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e62064fbdffb4954a3a1227efd6327ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:23.836292 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7296727c5b2f4c95ac3ff62818bd84f2" } modify_external_catalogs: true
I20251026 02:56:23.836593 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7296727c5b2f4c95ac3ff62818bd84f2" } modify_external_catalogs: true
I20251026 02:56:23.836853 2375 tablet_bootstrap.cc:492] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.837020 2375 ts_tablet_manager.cc:1403] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.093s user 0.007s sys 0.003s
I20251026 02:56:23.837009 2377 ts_tablet_manager.cc:1929] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.837139 2377 log.cc:1199] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/2ee0a408253e4f2b942c510ae64c703e
I20251026 02:56:23.837551 2377 ts_tablet_manager.cc:1950] T 2ee0a408253e4f2b942c510ae64c703e P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:23.837960 2375 raft_consensus.cc:359] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.838153 2375 raft_consensus.cc:385] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.838233 2375 raft_consensus.cc:740] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.838532 2375 consensus_queue.cc:260] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.840898 2375 ts_tablet_manager.cc:1434] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251026 02:56:23.841203 1658 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]). Expected version 2 got 0
I20251026 02:56:23.841532 2375 tablet_bootstrap.cc:492] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.845080 2375 tablet_bootstrap.cc:654] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.845317 1658 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]). Expected version 2 got 0
W20251026 02:56:23.849330 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 84ad6c72a8704c148411881760848df4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84ad6c72a8704c148411881760848df4 already in progress: creating tablet
I20251026 02:56:23.849921 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 2ee0a408253e4f2b942c510ae64c703e (table default.Table_c550a959e98a4e22853deca526de3eae [id=47bcd084e7fa4dec9f23cef12d10aa46]) successfully deleted
W20251026 02:56:23.850136 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 already in progress: creating tablet
W20251026 02:56:23.850320 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 already in progress: creating tablet
I20251026 02:56:23.851192 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet 7176cbe8f64546b7871d047c0e0e5b2a from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:23.851833 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9db88c5b7ec24571b75504849c925cfb" } new_table_name: "default.Table_8a12471ce48e486fa7959732288e0ff4" modify_external_catalogs: true
I20251026 02:56:23.853000 2375 tablet_bootstrap.cc:492] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.853150 2375 ts_tablet_manager.cc:1403] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20251026 02:56:23.854282 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b22d8322842a4ff6ad61fc3a8ad64f62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:23.854683 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4694119234644b988ca39446aa53f83f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.854856 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b22d8322842a4ff6ad61fc3a8ad64f62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.855391 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_be7d5ddd5e4e4b15862de09dc573f7f9" } modify_external_catalogs: true
I20251026 02:56:23.855611 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_be7d5ddd5e4e4b15862de09dc573f7f9" } modify_external_catalogs: true
I20251026 02:56:23.856734 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b389bf6a8e5d4ca39fb2c59f8103652c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.857226 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b389bf6a8e5d4ca39fb2c59f8103652c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.858402 2375 raft_consensus.cc:359] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.858464 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_80ca368a0ade4ac2b090b31b2ab56f12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:23.858639 2375 raft_consensus.cc:385] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.858733 2375 raft_consensus.cc:740] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
W20251026 02:56:23.858912 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80ca368a0ade4ac2b090b31b2ab56f12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.858994 2375 consensus_queue.cc:260] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.861584 2375 ts_tablet_manager.cc:1434] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.008s user 0.002s sys 0.002s
I20251026 02:56:23.862145 2375 tablet_bootstrap.cc:492] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.865633 2375 tablet_bootstrap.cc:654] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:23.855409 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4694119234644b988ca39446aa53f83f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.872942 2375 tablet_bootstrap.cc:492] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.873179 2375 ts_tablet_manager.cc:1403] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20251026 02:56:23.874497 2375 raft_consensus.cc:359] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.874784 2375 raft_consensus.cc:385] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.874881 2375 raft_consensus.cc:740] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.875247 2375 consensus_queue.cc:260] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.877182 2375 ts_tablet_manager.cc:1434] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251026 02:56:23.877628 2375 tablet_bootstrap.cc:492] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.880455 2375 tablet_bootstrap.cc:654] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.890565 2375 tablet_bootstrap.cc:492] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.890792 2375 ts_tablet_manager.cc:1403] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20251026 02:56:23.891916 2375 raft_consensus.cc:359] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.892181 2375 raft_consensus.cc:385] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.892340 2375 raft_consensus.cc:740] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.892699 2375 consensus_queue.cc:260] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.894731 2375 ts_tablet_manager.cc:1434] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251026 02:56:23.895136 2375 tablet_bootstrap.cc:492] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.898233 2375 tablet_bootstrap.cc:654] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.908846 2375 tablet_bootstrap.cc:492] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.909065 2375 ts_tablet_manager.cc:1403] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20251026 02:56:23.910012 2375 raft_consensus.cc:359] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.910264 2375 raft_consensus.cc:385] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.910362 2375 raft_consensus.cc:740] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.913358 2375 consensus_queue.cc:260] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.914935 2375 ts_tablet_manager.cc:1434] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20251026 02:56:23.915280 2375 tablet_bootstrap.cc:492] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.917354 2375 tablet_bootstrap.cc:654] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:23.943111 2375 tablet_bootstrap.cc:492] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:23.943367 2375 ts_tablet_manager.cc:1403] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.015s
I20251026 02:56:23.944470 2375 raft_consensus.cc:359] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.944727 2375 raft_consensus.cc:385] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:23.944841 2375 raft_consensus.cc:740] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:23.945103 2375 consensus_queue.cc:260] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:23.948028 2372 ts_tablet_manager.cc:1916] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:23.951505 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a70d70abc3a34dac8cf9a747903fa78c consensus state has no leader
W20251026 02:56:23.953579 1647 consensus_peers.cc:597] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 -> Peer 751b719b75ce4bc2a88a19091230ecb7 (127.1.150.60:33729): Couldn't send request to peer 751b719b75ce4bc2a88a19091230ecb7. Status: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251026 02:56:23.956311 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_cd142bf98f974283b7a2f9c1f1f27ef6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:23.957492 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet e0101098df05462bb26b3c187b259c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:23.958025 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd142bf98f974283b7a2f9c1f1f27ef6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.958230 2376 ts_tablet_manager.cc:1916] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:23.960227 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7176cbe8f64546b7871d047c0e0e5b2a
I20251026 02:56:23.961150 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_caee34582bea4cd1970c4cc59078f9d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:23.961542 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_542a136f8cdd4d32b2e1e6fbf7bcc448"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.961925 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_caee34582bea4cd1970c4cc59078f9d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:23.962088 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_542a136f8cdd4d32b2e1e6fbf7bcc448 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.962463 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet 0c647782049243d6bcd595f1a10d89c6 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:23.961953 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1972d81a4c494b7c97894d4991fed574" } modify_external_catalogs: true
I20251026 02:56:23.963649 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1972d81a4c494b7c97894d4991fed574" } modify_external_catalogs: true
W20251026 02:56:23.965605 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 049a915b35224cdaa38df77c734a0f01 has no consensus state
I20251026 02:56:23.966012 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9c97bccbf4cb4d15a66dbb8130acb6d4" } modify_external_catalogs: true
I20251026 02:56:23.966163 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9c97bccbf4cb4d15a66dbb8130acb6d4" } modify_external_catalogs: true
I20251026 02:56:23.966373 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_aeec0477dae74164b120683f450c5b94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.966869 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aeec0477dae74164b120683f450c5b94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.970366 2372 ts_tablet_manager.cc:1929] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.970578 2372 log.cc:1199] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/07280105a2b74ec68e27982414b12902
I20251026 02:56:23.971050 2372 ts_tablet_manager.cc:1950] T 07280105a2b74ec68e27982414b12902 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:23.972473 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 07280105a2b74ec68e27982414b12902 (table default.Table_81ca5851f387446ead27f921baefbe9a [id=6740f80827434cc393266a57238cdbdc]) successfully deleted
I20251026 02:56:23.973822 2376 ts_tablet_manager.cc:1929] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:23.974220 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 7176cbe8f64546b7871d047c0e0e5b2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.974761 2376 log.cc:1199] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/c0f83d631faa468f828e54e97987a00b
I20251026 02:56:23.975224 2376 ts_tablet_manager.cc:1950] T c0f83d631faa468f828e54e97987a00b P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:23.976380 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7439c051ca634fbe8b511d47ccede2ff
I20251026 02:56:23.977975 2375 ts_tablet_manager.cc:1434] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20251026 02:56:23.978822 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fbe94f0457844195a37ef907c6b74182" } new_table_name: "default.Table_9cfeb6e20450444da7688fe216f7a5c0" modify_external_catalogs: true
I20251026 02:56:23.980363 2375 tablet_bootstrap.cc:492] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:23.980585 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:23.981565 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7845ebd3e9ca4897b8fb19a48a010315
I20251026 02:56:23.981938 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6ce7e6a26747479bb5b4a5b04d37105a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.982502 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ce7e6a26747479bb5b4a5b04d37105a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.982947 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_03bc36a9fd184c28abd1caf6e5ec4f7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:23.983461 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03bc36a9fd184c28abd1caf6e5ec4f7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:23.984654 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 7439c051ca634fbe8b511d47ccede2ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.987167 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e89c69e3b58458aad10beaeb5128d60 with delete_type TABLET_DATA_DELETED (replaced by tablet a821ed8a686c452a975acb96b5c74778) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.993117 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_af080a4fb1da461eb6bf487aa8e54990" } new_table_name: "default.Table_14cbc77849c244ae98b9a3afc5a8fdc0" modify_external_catalogs: true
I20251026 02:56:23.995153 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 2487bf7d40f34922aa2fd165488342f6 with delete_type TABLET_DATA_DELETED (replaced by tablet a111848706b743c9beb2fbd21f59d7f3) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.995822 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.997010 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_99b42c4a4a874b68bb4beba4c08cdeb0" } new_table_name: "default.Table_65854c69bc7b4af88efa140b222d480b" modify_external_catalogs: true
I20251026 02:56:23.998252 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet b108f0d3dae84fab93bff012400c4a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:23.998539 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 7845ebd3e9ca4897b8fb19a48a010315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.001667 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet 829a6044a2da44ff90acdebca6d5b18d from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:24.002130 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ec035b1caac450d9ab55e74a061ae38 consensus state has no leader
I20251026 02:56:24.002452 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet 7176cbe8f64546b7871d047c0e0e5b2a from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:24.001462 2375 tablet_bootstrap.cc:654] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.002123 2372 tablet_replica.cc:333] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.004259 2372 raft_consensus.cc:2243] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.004475 2372 raft_consensus.cc:2272] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.004845 1952 tablet_service.cc:1552] Processing DeleteTablet for tablet 7176cbe8f64546b7871d047c0e0e5b2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.006007 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 7176cbe8f64546b7871d047c0e0e5b2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.006999 2372 ts_tablet_manager.cc:1916] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.008973 1952 tablet_service.cc:1552] Processing DeleteTablet for tablet 7439c051ca634fbe8b511d47ccede2ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:24.009207 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22e4a433801a4cb58062d3e76c760b69 consensus state has no leader
I20251026 02:56:24.010001 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 7439c051ca634fbe8b511d47ccede2ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.011991 1952 tablet_service.cc:1552] Processing DeleteTablet for tablet 7845ebd3e9ca4897b8fb19a48a010315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.012979 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 7845ebd3e9ca4897b8fb19a48a010315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.015615 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet c0f83d631faa468f828e54e97987a00b (table default.Table_4578a4ce81ea42cd878859a8167babff [id=f914c9974ac44d3f84c3d312649fe255]) successfully deleted
I20251026 02:56:24.017406 2378 tablet_bootstrap.cc:492] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:24.017602 2378 ts_tablet_manager.cc:1403] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.262s user 0.022s sys 0.037s
W20251026 02:56:24.016044 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81530bbf4a2d4e908add683475e594fb consensus state has no leader
I20251026 02:56:24.018532 2378 raft_consensus.cc:359] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.018739 2378 raft_consensus.cc:385] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.018826 2378 raft_consensus.cc:740] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.018854 2381 tablet_bootstrap.cc:492] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:24.019153 2381 ts_tablet_manager.cc:1403] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.285s user 0.072s sys 0.079s
I20251026 02:56:24.019191 2378 consensus_queue.cc:260] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.021245 2381 raft_consensus.cc:359] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.021626 2381 raft_consensus.cc:385] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.021745 2381 raft_consensus.cc:740] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.022091 2381 consensus_queue.cc:260] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.023018 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 115ccccf16b74efc9f24b607cf0c74d1 (DEFAULT_TABLE table=default.Table_7b6687267a1a4b2da7e02949e7b4c081 [id=ad981d3883cc4ee5b77defff9651ef31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.023551 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 115ccccf16b74efc9f24b607cf0c74d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.027479 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 0efdcc36ff5b47379e0a76512e05d379 (DEFAULT_TABLE table=default.Table_1972d81a4c494b7c97894d4991fed574 [id=d9f8b0fbb3ed416fb380a5717fb29dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.027909 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0efdcc36ff5b47379e0a76512e05d379. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.028195 2375 tablet_bootstrap.cc:492] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.028388 2375 ts_tablet_manager.cc:1403] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.000s
I20251026 02:56:24.029307 2375 raft_consensus.cc:359] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.029466 2375 raft_consensus.cc:385] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.029528 2375 raft_consensus.cc:740] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.029773 2375 consensus_queue.cc:260] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.029891 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 049a915b35224cdaa38df77c734a0f01 (DEFAULT_TABLE table=default.Table_8a12471ce48e486fa7959732288e0ff4 [id=f1f9d2ad39b34abbbe989bc3fac62b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.030223 2375 ts_tablet_manager.cc:1434] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:24.030329 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 049a915b35224cdaa38df77c734a0f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.030531 2375 tablet_bootstrap.cc:492] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.030825 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 4a5343ba771a4e8ea2f467c4c619c1f9
I20251026 02:56:24.031842 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0efdcc36ff5b47379e0a76512e05d379
I20251026 02:56:24.032529 2375 tablet_bootstrap.cc:654] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.032522 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee
I20251026 02:56:24.032924 1656 catalog_manager.cc:6719] ReplaceTablet: tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee deleted and replaced by tablet 634d913c98c842a49d97a54f23752d26
I20251026 02:56:24.033088 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.034382 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0c647782049243d6bcd595f1a10d89c6
I20251026 02:56:24.034541 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 548a7f9628ac4623b2c2f723f83c1e45 (DEFAULT_TABLE table=default.Table_fafb3f2678b846e3bf05a461dde63011 [id=d9a6b0b6f3e246a3aa8fbd1167205264]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.034787 1657 catalog_manager.cc:6719] ReplaceTablet: tablet 0c647782049243d6bcd595f1a10d89c6 deleted and replaced by tablet 5caeb11322fe48348ebb523967152187
I20251026 02:56:24.034910 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 548a7f9628ac4623b2c2f723f83c1e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.035578 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee with delete_type TABLET_DATA_DELETED (replaced by tablet 634d913c98c842a49d97a54f23752d26) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.035817 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c647782049243d6bcd595f1a10d89c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 5caeb11322fe48348ebb523967152187) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.036044 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 0efdcc36ff5b47379e0a76512e05d379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.036432 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 03da415f3a944a9a9679b3a45295ce35 (DEFAULT_TABLE table=default.Table_828c16a96a47489f835a891f34d707b1 [id=432234d273ca4c9ea53c7a078c274637]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.036782 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03da415f3a944a9a9679b3a45295ce35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.039127 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 122de8616b8f4fcf88971d5738c47e82 (DEFAULT_TABLE table=default.Table_719b11bde46041c5a20750db14381807 [id=4912ce6ea2304b5a9c21dd91a7cbd56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.039548 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122de8616b8f4fcf88971d5738c47e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.040863 2089 tablet_service.cc:1505] Processing CreateTablet for tablet 94886ad6d6bb45f39261725169774ac9 (DEFAULT_TABLE table=default.Table_d23ac2c9657b4adeaac9545d5bc9ae41 [id=9e897d33e6374ad4bc3c8acaae512645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.041532 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94886ad6d6bb45f39261725169774ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.041656 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 829a6044a2da44ff90acdebca6d5b18d
I20251026 02:56:24.042145 1659 catalog_manager.cc:6719] ReplaceTablet: tablet 829a6044a2da44ff90acdebca6d5b18d deleted and replaced by tablet 0de06ea2b0544402a979d9ddc487860f
I20251026 02:56:24.042318 2378 ts_tablet_manager.cc:1434] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.025s user 0.001s sys 0.002s
I20251026 02:56:24.042560 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 1e56b259a314422aba35eb129021bbeb (DEFAULT_TABLE table=default.Table_25607539f64440cb8ff2c24a0a3a3bdd [id=5dbfb5d912ba433baa46bd52bbf6dbee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.044672 2378 tablet_bootstrap.cc:492] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:24.045326 2381 ts_tablet_manager.cc:1434] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20251026 02:56:24.046270 2381 tablet_bootstrap.cc:492] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:24.042003 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7176cbe8f64546b7871d047c0e0e5b2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7176cbe8f64546b7871d047c0e0e5b2a already in progress: creating tablet
W20251026 02:56:24.047609 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7439c051ca634fbe8b511d47ccede2ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7439c051ca634fbe8b511d47ccede2ff already in progress: creating tablet
W20251026 02:56:24.047875 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7845ebd3e9ca4897b8fb19a48a010315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7845ebd3e9ca4897b8fb19a48a010315 already in progress: creating tablet
I20251026 02:56:24.048852 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e56b259a314422aba35eb129021bbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.049372 2378 tablet_bootstrap.cc:654] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.051651 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 4a5b68772a66402bbb0b7e00bf96fcc0 (DEFAULT_TABLE table=default.Table_1e459a892e604d56aeb212131b5a624f [id=8f39b9747e16469eb211e44a69a276e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.052073 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 115ccccf16b74efc9f24b607cf0c74d1 (DEFAULT_TABLE table=default.Table_7b6687267a1a4b2da7e02949e7b4c081 [id=ad981d3883cc4ee5b77defff9651ef31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.052522 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 115ccccf16b74efc9f24b607cf0c74d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.052793 2381 tablet_bootstrap.cc:654] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.051322 2098 tablet_service.cc:1505] Processing CreateTablet for tablet b2bfd9b55c484ef8a9c6345d56f794fb (DEFAULT_TABLE table=default.Table_30e98b21e65d443097e2fc01f96f4780 [id=0d9be16d1aec415a8f9a2433f8d7ac6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.054230 2090 tablet_service.cc:1505] Processing CreateTablet for tablet d90e62ae86b149b291dc29cf7442c6bc (DEFAULT_TABLE table=default.Table_e2a5e05b70f841cfb0065041b39e89cb [id=e1c065cb2f704594873006d80a406df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.054721 2093 tablet_service.cc:1505] Processing CreateTablet for tablet ced7ad6a601d432cbc13b3199a0506e4 (DEFAULT_TABLE table=default.Table_3316a969a3654fdf853bca2d3d666aa4 [id=4e7fddec9b124b32b7786ea15111878b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.055397 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 829a6044a2da44ff90acdebca6d5b18d with delete_type TABLET_DATA_DELETED (replaced by tablet 0de06ea2b0544402a979d9ddc487860f) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.055673 2378 tablet_bootstrap.cc:492] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:24.055848 2378 ts_tablet_manager.cc:1403] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20251026 02:56:24.056108 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet c0f83d631faa468f828e54e97987a00b with delete_type TABLET_DATA_DELETED (replaced by tablet 9156314f98ac4ed9981f6f13fc585043) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.056391 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a5b68772a66402bbb0b7e00bf96fcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.056797 2378 raft_consensus.cc:359] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.057008 2378 raft_consensus.cc:385] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.057106 2378 raft_consensus.cc:740] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.057404 2378 consensus_queue.cc:260] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.059980 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ced7ad6a601d432cbc13b3199a0506e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.060573 2378 ts_tablet_manager.cc:1434] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251026 02:56:24.061968 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 115ccccf16b74efc9f24b607cf0c74d1 (DEFAULT_TABLE table=default.Table_7b6687267a1a4b2da7e02949e7b4c081 [id=ad981d3883cc4ee5b77defff9651ef31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.062474 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 115ccccf16b74efc9f24b607cf0c74d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.063019 2378 tablet_bootstrap.cc:492] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:24.064293 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2bfd9b55c484ef8a9c6345d56f794fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.066612 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d90e62ae86b149b291dc29cf7442c6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.070060 2378 tablet_bootstrap.cc:654] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.075258 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet b108f0d3dae84fab93bff012400c4a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.075343 2381 tablet_bootstrap.cc:492] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:24.075489 2381 ts_tablet_manager.cc:1403] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.008s
I20251026 02:56:24.075555 2376 tablet_replica.cc:333] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:24.075760 2376 raft_consensus.cc:2243] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.075937 2376 raft_consensus.cc:2272] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.076305 2381 raft_consensus.cc:359] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.076509 2381 raft_consensus.cc:385] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.076608 2381 raft_consensus.cc:740] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.076785 2376 ts_tablet_manager.cc:1916] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.076854 2381 consensus_queue.cc:260] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.077651 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet ad55040f0a2b4cfa894d1bd058d4d69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.077908 2377 tablet_replica.cc:333] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:24.078089 2377 raft_consensus.cc:2243] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.078236 2377 raft_consensus.cc:2272] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.078440 2405 raft_consensus.cc:493] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:24.078555 2405 raft_consensus.cc:515] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.078706 2381 ts_tablet_manager.cc:1434] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:24.079015 2381 tablet_bootstrap.cc:492] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:24.079209 2405 leader_election.cc:290] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:24.080822 2381 tablet_bootstrap.cc:654] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.080855 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2bfbe624075463f8b90ed4ec66b784c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:24.081497 2194 leader_election.cc:343] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.082175 2377 ts_tablet_manager.cc:1916] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.082312 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2bfbe624075463f8b90ed4ec66b784c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:24.083269 2120 raft_consensus.cc:2468] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:24.084228 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 548a7f9628ac4623b2c2f723f83c1e45 (DEFAULT_TABLE table=default.Table_fafb3f2678b846e3bf05a461dde63011 [id=d9a6b0b6f3e246a3aa8fbd1167205264]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.084635 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 548a7f9628ac4623b2c2f723f83c1e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.086959 2378 tablet_bootstrap.cc:492] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:24.087136 2378 ts_tablet_manager.cc:1403] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20251026 02:56:24.087941 2378 raft_consensus.cc:359] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.088157 2378 raft_consensus.cc:385] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.088267 2378 raft_consensus.cc:740] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.088567 2378 consensus_queue.cc:260] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.088608 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 548a7f9628ac4623b2c2f723f83c1e45 (DEFAULT_TABLE table=default.Table_fafb3f2678b846e3bf05a461dde63011 [id=d9a6b0b6f3e246a3aa8fbd1167205264]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.089030 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 548a7f9628ac4623b2c2f723f83c1e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.089028 2195 leader_election.cc:304] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:24.089457 2405 raft_consensus.cc:2804] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:24.089561 2405 raft_consensus.cc:493] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:24.089643 2405 raft_consensus.cc:3060] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:24.091327 2378 ts_tablet_manager.cc:1434] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:24.092413 2506 raft_consensus.cc:493] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:24.092613 2506 raft_consensus.cc:515] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.093338 2506 leader_election.cc:290] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:24.093562 2378 tablet_bootstrap.cc:492] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:24.095155 2053 leader_election.cc:343] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: c0f83d631faa468f828e54e97987a00b
W20251026 02:56:24.095395 2053 leader_election.cc:343] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: c0f83d631faa468f828e54e97987a00b
I20251026 02:56:24.095534 2053 leader_election.cc:304] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:24.095801 2506 raft_consensus.cc:2749] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:24.095669 2405 raft_consensus.cc:515] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.096824 2378 tablet_bootstrap.cc:654] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.097240 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2bfbe624075463f8b90ed4ec66b784c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:24.097437 2120 raft_consensus.cc:3060] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:24.100194 2376 ts_tablet_manager.cc:1929] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.100376 2376 log.cc:1199] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/b108f0d3dae84fab93bff012400c4a2a
I20251026 02:56:24.100739 2376 ts_tablet_manager.cc:1950] T b108f0d3dae84fab93bff012400c4a2a P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:24.100842 2405 leader_election.cc:290] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:24.101298 2120 raft_consensus.cc:2468] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:24.101739 2195 leader_election.cc:304] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:24.102090 2405 raft_consensus.cc:2804] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:24.102772 2405 raft_consensus.cc:697] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:24.104504 2405 consensus_queue.cc:237] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.106900 2372 ts_tablet_manager.cc:1929] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.107079 2372 log.cc:1199] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/e0101098df05462bb26b3c187b259c08
I20251026 02:56:24.107498 2372 ts_tablet_manager.cc:1950] T e0101098df05462bb26b3c187b259c08 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.107506 2377 ts_tablet_manager.cc:1929] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.107765 2377 log.cc:1199] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/ad55040f0a2b4cfa894d1bd058d4d69f
I20251026 02:56:24.108249 2377 ts_tablet_manager.cc:1950] T ad55040f0a2b4cfa894d1bd058d4d69f P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:24.100942 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2bfbe624075463f8b90ed4ec66b784c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:24.109747 2194 leader_election.cc:343] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.114841 2372 tablet_replica.cc:333] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.115118 2372 raft_consensus.cc:2243] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.115355 2372 raft_consensus.cc:2272] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.116242 2372 ts_tablet_manager.cc:1916] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.121263 2087 tablet_service.cc:1505] Processing CreateTablet for tablet f85c2745a371455a8dda67f31a19c9d5 (DEFAULT_TABLE table=default.Table_542a136f8cdd4d32b2e1e6fbf7bcc448 [id=f802cc99a8ec439a85a18e9b9a2826e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.122053 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 0efdcc36ff5b47379e0a76512e05d379 (DEFAULT_TABLE table=default.Table_1972d81a4c494b7c97894d4991fed574 [id=d9f8b0fbb3ed416fb380a5717fb29dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.122357 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 863ce9e1adb444b7b1399b968ea79bb4 (DEFAULT_TABLE table=default.Table_03bc36a9fd184c28abd1caf6e5ec4f7a [id=ee13c495b56c48649ceacc8e1122efb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.122570 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0efdcc36ff5b47379e0a76512e05d379. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.123476 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 63a63b796191465aab9a63d3ffb0554d (DEFAULT_TABLE table=default.Table_b389bf6a8e5d4ca39fb2c59f8103652c [id=d27916c5ae83463e868d14a5d22ae281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.123674 2085 tablet_service.cc:1505] Processing CreateTablet for tablet a08efd37968846d2958e54831488235c (DEFAULT_TABLE table=default.Table_80ca368a0ade4ac2b090b31b2ab56f12 [id=49dd7ccfa92946bebeaa4c178138f75a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.126308 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet e0101098df05462bb26b3c187b259c08 (table default.Table_bcf0aac1b41747e9a94a97b8430fa081 [id=df8666bd175f43ceb47649f67a3d865e]) successfully deleted
I20251026 02:56:24.126449 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 352a284902c94f088e2e6ae006162db0 (DEFAULT_TABLE table=default.Table_4694119234644b988ca39446aa53f83f [id=414375003cd44b5c8c9f0249b339c28a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:24.126848 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7176cbe8f64546b7871d047c0e0e5b2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7176cbe8f64546b7871d047c0e0e5b2a already in progress: creating tablet
W20251026 02:56:24.127266 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7439c051ca634fbe8b511d47ccede2ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7439c051ca634fbe8b511d47ccede2ff already in progress: creating tablet
I20251026 02:56:24.127485 2378 tablet_bootstrap.cc:492] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:24.127660 2378 ts_tablet_manager.cc:1403] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.007s
I20251026 02:56:24.131570 1659 catalog_manager.cc:5649] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:24.132023 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]). Expected version 2 got 0
I20251026 02:56:24.132709 2378 raft_consensus.cc:359] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.132910 2378 raft_consensus.cc:385] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.133031 2378 raft_consensus.cc:740] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.133278 2378 consensus_queue.cc:260] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.133814 2378 ts_tablet_manager.cc:1434] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251026 02:56:24.135172 2378 tablet_bootstrap.cc:492] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:24.134240 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 0efdcc36ff5b47379e0a76512e05d379 (DEFAULT_TABLE table=default.Table_1972d81a4c494b7c97894d4991fed574 [id=d9f8b0fbb3ed416fb380a5717fb29dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.136343 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0efdcc36ff5b47379e0a76512e05d379. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.140520 2378 tablet_bootstrap.cc:654] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.152947 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.154389 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.155225 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 0efdcc36ff5b47379e0a76512e05d379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.156603 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 0efdcc36ff5b47379e0a76512e05d379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.156939 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee with delete_type TABLET_DATA_DELETED (replaced by tablet 634d913c98c842a49d97a54f23752d26) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.157327 2092 tablet_service.cc:1505] Processing CreateTablet for tablet f6d6f024d8b943d5acf69b64bc7c7214 (DEFAULT_TABLE table=default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.157974 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee with delete_type TABLET_DATA_DELETED (replaced by tablet 634d913c98c842a49d97a54f23752d26) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.158775 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f85c2745a371455a8dda67f31a19c9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.159067 2375 tablet_bootstrap.cc:492] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.159109 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63a63b796191465aab9a63d3ffb0554d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.159227 2375 ts_tablet_manager.cc:1403] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.129s user 0.004s sys 0.000s
I20251026 02:56:24.159364 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a08efd37968846d2958e54831488235c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.159399 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c647782049243d6bcd595f1a10d89c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 5caeb11322fe48348ebb523967152187) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.159610 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352a284902c94f088e2e6ae006162db0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.159832 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 863ce9e1adb444b7b1399b968ea79bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.160261 2375 raft_consensus.cc:359] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.160470 2375 raft_consensus.cc:385] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.160578 2375 raft_consensus.cc:740] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.160540 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 03da415f3a944a9a9679b3a45295ce35 (DEFAULT_TABLE table=default.Table_828c16a96a47489f835a891f34d707b1 [id=432234d273ca4c9ea53c7a078c274637]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.160809 2375 consensus_queue.cc:260] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.161038 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03da415f3a944a9a9679b3a45295ce35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.161367 2375 ts_tablet_manager.cc:1434] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:24.161688 2375 tablet_bootstrap.cc:492] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.164638 2375 tablet_bootstrap.cc:654] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.168833 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c647782049243d6bcd595f1a10d89c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 5caeb11322fe48348ebb523967152187) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.171064 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 03da415f3a944a9a9679b3a45295ce35 (DEFAULT_TABLE table=default.Table_828c16a96a47489f835a891f34d707b1 [id=432234d273ca4c9ea53c7a078c274637]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.171638 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03da415f3a944a9a9679b3a45295ce35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.181352 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 049a915b35224cdaa38df77c734a0f01 (DEFAULT_TABLE table=default.Table_8a12471ce48e486fa7959732288e0ff4 [id=f1f9d2ad39b34abbbe989bc3fac62b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.182030 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 049a915b35224cdaa38df77c734a0f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.182509 2375 tablet_bootstrap.cc:492] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.182693 2375 ts_tablet_manager.cc:1403] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20251026 02:56:24.183508 2375 raft_consensus.cc:359] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.183815 2375 raft_consensus.cc:385] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.183907 2375 raft_consensus.cc:740] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.184195 2375 consensus_queue.cc:260] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.185480 2375 ts_tablet_manager.cc:1434] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251026 02:56:24.186070 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 049a915b35224cdaa38df77c734a0f01 (DEFAULT_TABLE table=default.Table_8a12471ce48e486fa7959732288e0ff4 [id=f1f9d2ad39b34abbbe989bc3fac62b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.186164 2375 tablet_bootstrap.cc:492] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.186731 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 049a915b35224cdaa38df77c734a0f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.188158 2375 tablet_bootstrap.cc:654] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.193962 2375 tablet_bootstrap.cc:492] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.194135 2375 ts_tablet_manager.cc:1403] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20251026 02:56:24.195000 2375 raft_consensus.cc:359] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.195206 2375 raft_consensus.cc:385] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.195304 2375 raft_consensus.cc:740] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.195538 2375 consensus_queue.cc:260] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.197089 2378 tablet_bootstrap.cc:492] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:24.197286 2378 ts_tablet_manager.cc:1403] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.016s
I20251026 02:56:24.198150 2378 raft_consensus.cc:359] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.198354 2378 raft_consensus.cc:385] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.198427 2378 raft_consensus.cc:740] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.198669 2378 consensus_queue.cc:260] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.199218 2378 ts_tablet_manager.cc:1434] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:24.199573 2378 tablet_bootstrap.cc:492] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:24.199626 2239 tablet_service.cc:1505] Processing CreateTablet for tablet d90e62ae86b149b291dc29cf7442c6bc (DEFAULT_TABLE table=default.Table_e2a5e05b70f841cfb0065041b39e89cb [id=e1c065cb2f704594873006d80a406df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.200052 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d90e62ae86b149b291dc29cf7442c6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.201566 1954 tablet_service.cc:1505] Processing CreateTablet for tablet d90e62ae86b149b291dc29cf7442c6bc (DEFAULT_TABLE table=default.Table_e2a5e05b70f841cfb0065041b39e89cb [id=e1c065cb2f704594873006d80a406df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.201974 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d90e62ae86b149b291dc29cf7442c6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.202394 2375 ts_tablet_manager.cc:1434] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251026 02:56:24.202726 2375 tablet_bootstrap.cc:492] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.207294 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 81530bbf4a2d4e908add683475e594fb (table default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]). Expected version 1 got 0
I20251026 02:56:24.207440 1962 tablet_service.cc:1505] Processing CreateTablet for tablet ced7ad6a601d432cbc13b3199a0506e4 (DEFAULT_TABLE table=default.Table_3316a969a3654fdf853bca2d3d666aa4 [id=4e7fddec9b124b32b7786ea15111878b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.207980 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ced7ad6a601d432cbc13b3199a0506e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.209304 2242 tablet_service.cc:1505] Processing CreateTablet for tablet ced7ad6a601d432cbc13b3199a0506e4 (DEFAULT_TABLE table=default.Table_3316a969a3654fdf853bca2d3d666aa4 [id=4e7fddec9b124b32b7786ea15111878b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.209803 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ced7ad6a601d432cbc13b3199a0506e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.212183 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet b435520cadb344edaa29975b4e19718a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.212364 2378 tablet_bootstrap.cc:654] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.212560 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 565255802f4f42ba8320a3ec80d2db8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.214700 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 634d913c98c842a49d97a54f23752d26 (DEFAULT_TABLE table=default.Table_d2dcec579c684b92a92d98fb59ee3da1 [id=4f7edfb6d75c4a0b94ca99bb9cfe14f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.215152 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 0de06ea2b0544402a979d9ddc487860f (DEFAULT_TABLE table=default.Table_6546379855b749b889279973db1820cd [id=b8be7adba1cd48ae979de506d892cc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.215744 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]). Expected version 2 got 0
I20251026 02:56:24.213147 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d6f024d8b943d5acf69b64bc7c7214. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.216574 2405 raft_consensus.cc:493] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:24.216823 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 634d913c98c842a49d97a54f23752d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.216760 2405 raft_consensus.cc:515] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.217468 2405 leader_election.cc:290] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:24.217535 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 122de8616b8f4fcf88971d5738c47e82 (DEFAULT_TABLE table=default.Table_719b11bde46041c5a20750db14381807 [id=4912ce6ea2304b5a9c21dd91a7cbd56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.218811 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81530bbf4a2d4e908add683475e594fb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:24.219067 2120 raft_consensus.cc:2468] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:24.219183 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81530bbf4a2d4e908add683475e594fb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:24.213013 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 5caeb11322fe48348ebb523967152187 (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.219784 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122de8616b8f4fcf88971d5738c47e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.219915 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5caeb11322fe48348ebb523967152187. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.215735 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 276bd3b97c97426daf802e1f57feb265 (DEFAULT_TABLE table=default.Table_cd142bf98f974283b7a2f9c1f1f27ef6 [id=e91358e10bab4589a6f4e815a85a5f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.224488 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 276bd3b97c97426daf802e1f57feb265. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.227221 2372 ts_tablet_manager.cc:1929] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.227356 2372 log.cc:1199] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/0e89c69e3b58458aad10beaeb5128d60
W20251026 02:56:24.227608 2194 leader_election.cc:343] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.227890 2195 leader_election.cc:304] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:24.228485 2405 raft_consensus.cc:2804] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:24.228602 2405 raft_consensus.cc:493] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:24.228724 2405 raft_consensus.cc:3060] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:24.232405 2375 tablet_bootstrap.cc:654] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.233343 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]). Expected version 2 got 0
I20251026 02:56:24.233906 2372 ts_tablet_manager.cc:1950] T 0e89c69e3b58458aad10beaeb5128d60 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.235134 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 0e89c69e3b58458aad10beaeb5128d60 (table default.Table_00dc4c49a19246889353df2157eebf15 [id=347d712aca4e477c89805846800fb824]) successfully deleted
I20251026 02:56:24.235498 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de06ea2b0544402a979d9ddc487860f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.235512 2405 raft_consensus.cc:515] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.216156 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 0fe0e06a86e241bc91a1db4219fe25bc (DEFAULT_TABLE table=default.Table_caee34582bea4cd1970c4cc59078f9d1 [id=0494cd534eb24adc94c907471cdb346a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.236292 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe0e06a86e241bc91a1db4219fe25bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.236795 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81530bbf4a2d4e908add683475e594fb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:24.237423 2194 leader_election.cc:343] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.237721 2378 tablet_bootstrap.cc:492] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:24.237882 2378 ts_tablet_manager.cc:1403] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.005s
I20251026 02:56:24.237916 2405 leader_election.cc:290] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:24.238772 2378 raft_consensus.cc:359] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.238976 2378 raft_consensus.cc:385] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.239070 2378 raft_consensus.cc:740] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.237704 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81530bbf4a2d4e908add683475e594fb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:24.238618 2372 tablet_replica.cc:333] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.239380 2120 raft_consensus.cc:3060] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:24.239320 2378 consensus_queue.cc:260] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.239406 2233 tablet_service.cc:1505] Processing CreateTablet for tablet b2bfd9b55c484ef8a9c6345d56f794fb (DEFAULT_TABLE table=default.Table_30e98b21e65d443097e2fc01f96f4780 [id=0d9be16d1aec415a8f9a2433f8d7ac6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.239476 2372 raft_consensus.cc:2243] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.239636 2372 raft_consensus.cc:2272] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.239797 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2bfd9b55c484ef8a9c6345d56f794fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.242022 2378 ts_tablet_manager.cc:1434] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:24.242314 2378 tablet_bootstrap.cc:492] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:24.244012 2378 tablet_bootstrap.cc:654] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.247336 2120 raft_consensus.cc:2468] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:24.248240 2195 leader_election.cc:304] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:24.249907 2405 raft_consensus.cc:2804] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:24.250908 2405 raft_consensus.cc:697] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:24.251417 2405 consensus_queue.cc:237] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.252563 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 122de8616b8f4fcf88971d5738c47e82 (DEFAULT_TABLE table=default.Table_719b11bde46041c5a20750db14381807 [id=4912ce6ea2304b5a9c21dd91a7cbd56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.253171 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122de8616b8f4fcf88971d5738c47e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.262120 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 4a5b68772a66402bbb0b7e00bf96fcc0 (DEFAULT_TABLE table=default.Table_1e459a892e604d56aeb212131b5a624f [id=8f39b9747e16469eb211e44a69a276e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.262702 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a5b68772a66402bbb0b7e00bf96fcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.269349 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 1e56b259a314422aba35eb129021bbeb (DEFAULT_TABLE table=default.Table_25607539f64440cb8ff2c24a0a3a3bdd [id=5dbfb5d912ba433baa46bd52bbf6dbee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.269728 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 4a5b68772a66402bbb0b7e00bf96fcc0 (DEFAULT_TABLE table=default.Table_1e459a892e604d56aeb212131b5a624f [id=8f39b9747e16469eb211e44a69a276e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.269994 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e56b259a314422aba35eb129021bbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.270134 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a5b68772a66402bbb0b7e00bf96fcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.272266 2381 tablet_bootstrap.cc:492] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:24.272972 1962 tablet_service.cc:1505] Processing CreateTablet for tablet b2bfd9b55c484ef8a9c6345d56f794fb (DEFAULT_TABLE table=default.Table_30e98b21e65d443097e2fc01f96f4780 [id=0d9be16d1aec415a8f9a2433f8d7ac6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.273595 2381 ts_tablet_manager.cc:1403] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.195s user 0.012s sys 0.026s
I20251026 02:56:24.273486 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2bfd9b55c484ef8a9c6345d56f794fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.278326 2372 ts_tablet_manager.cc:1916] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.272572 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 1e56b259a314422aba35eb129021bbeb (DEFAULT_TABLE table=default.Table_25607539f64440cb8ff2c24a0a3a3bdd [id=5dbfb5d912ba433baa46bd52bbf6dbee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.280673 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e56b259a314422aba35eb129021bbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.280918 2381 raft_consensus.cc:359] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.281147 2381 raft_consensus.cc:385] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.281224 2381 raft_consensus.cc:740] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.281483 2381 consensus_queue.cc:260] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.282004 2381 ts_tablet_manager.cc:1434] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.008s user 0.000s sys 0.000s
I20251026 02:56:24.282750 2381 tablet_bootstrap.cc:492] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:24.284078 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 829a6044a2da44ff90acdebca6d5b18d with delete_type TABLET_DATA_DELETED (replaced by tablet 0de06ea2b0544402a979d9ddc487860f) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:24.284838 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7176cbe8f64546b7871d047c0e0e5b2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7176cbe8f64546b7871d047c0e0e5b2a already in progress: creating tablet
W20251026 02:56:24.285120 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7439c051ca634fbe8b511d47ccede2ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7439c051ca634fbe8b511d47ccede2ff already in progress: creating tablet
W20251026 02:56:24.285362 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7845ebd3e9ca4897b8fb19a48a010315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7845ebd3e9ca4897b8fb19a48a010315 already in progress: creating tablet
I20251026 02:56:24.285676 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet ad55040f0a2b4cfa894d1bd058d4d69f (table default.Table_0da51595453549b3b7e077f7b7ff3ec7 [id=53cd517e68e64597ad51d85d16f0399d]) successfully deleted
W20251026 02:56:24.286036 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a5343ba771a4e8ea2f467c4c619c1f9 already in progress: creating tablet
W20251026 02:56:24.286263 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0efdcc36ff5b47379e0a76512e05d379 because the tablet was not found. No further retry: Not found: Tablet not found: 0efdcc36ff5b47379e0a76512e05d379
W20251026 02:56:24.286492 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee already in progress: creating tablet
W20251026 02:56:24.286706 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0c647782049243d6bcd595f1a10d89c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c647782049243d6bcd595f1a10d89c6 already in progress: creating tablet
I20251026 02:56:24.288368 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 94886ad6d6bb45f39261725169774ac9 (DEFAULT_TABLE table=default.Table_d23ac2c9657b4adeaac9545d5bc9ae41 [id=9e897d33e6374ad4bc3c8acaae512645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.288489 1959 tablet_service.cc:1505] Processing CreateTablet for tablet f6d6f024d8b943d5acf69b64bc7c7214 (DEFAULT_TABLE table=default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.288852 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d6f024d8b943d5acf69b64bc7c7214. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.288895 1955 tablet_service.cc:1505] Processing CreateTablet for tablet f85c2745a371455a8dda67f31a19c9d5 (DEFAULT_TABLE table=default.Table_542a136f8cdd4d32b2e1e6fbf7bcc448 [id=f802cc99a8ec439a85a18e9b9a2826e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.289260 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f85c2745a371455a8dda67f31a19c9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.290200 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 94886ad6d6bb45f39261725169774ac9 (DEFAULT_TABLE table=default.Table_d23ac2c9657b4adeaac9545d5bc9ae41 [id=9e897d33e6374ad4bc3c8acaae512645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.290639 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94886ad6d6bb45f39261725169774ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.291343 2372 ts_tablet_manager.cc:1929] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.291483 2372 log.cc:1199] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/2487bf7d40f34922aa2fd165488342f6
I20251026 02:56:24.291711 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.291844 2372 ts_tablet_manager.cc:1950] T 2487bf7d40f34922aa2fd165488342f6 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.292065 2376 tablet_replica.cc:333] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:24.292316 2376 raft_consensus.cc:2243] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.292516 2376 raft_consensus.cc:2272] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.293062 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 2487bf7d40f34922aa2fd165488342f6 (table default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]) successfully deleted
I20251026 02:56:24.293418 2372 tablet_replica.cc:333] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.293826 2238 tablet_service.cc:1505] Processing CreateTablet for tablet f6d6f024d8b943d5acf69b64bc7c7214 (DEFAULT_TABLE table=default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.294530 2372 raft_consensus.cc:2243] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.294514 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 565255802f4f42ba8320a3ec80d2db8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.294767 2372 raft_consensus.cc:2272] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.295082 2234 tablet_service.cc:1505] Processing CreateTablet for tablet f85c2745a371455a8dda67f31a19c9d5 (DEFAULT_TABLE table=default.Table_542a136f8cdd4d32b2e1e6fbf7bcc448 [id=f802cc99a8ec439a85a18e9b9a2826e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.295382 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d6f024d8b943d5acf69b64bc7c7214. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.295624 2372 ts_tablet_manager.cc:1916] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.296147 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f85c2745a371455a8dda67f31a19c9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.298482 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 829a6044a2da44ff90acdebca6d5b18d with delete_type TABLET_DATA_DELETED (replaced by tablet 0de06ea2b0544402a979d9ddc487860f) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.299860 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 863ce9e1adb444b7b1399b968ea79bb4 (DEFAULT_TABLE table=default.Table_03bc36a9fd184c28abd1caf6e5ec4f7a [id=ee13c495b56c48649ceacc8e1122efb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.300293 2376 ts_tablet_manager.cc:1916] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.305814 2377 tablet_replica.cc:333] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:24.306041 2377 raft_consensus.cc:2243] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.306131 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 2524ee0c964345459c3e62f1e39b8f9e (DEFAULT_TABLE table=default.Table_6ce7e6a26747479bb5b4a5b04d37105a [id=91b4b1a8162441299c680e7c202a96d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.306519 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2524ee0c964345459c3e62f1e39b8f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.307838 2377 raft_consensus.cc:2272] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.310097 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 863ce9e1adb444b7b1399b968ea79bb4 (DEFAULT_TABLE table=default.Table_03bc36a9fd184c28abd1caf6e5ec4f7a [id=ee13c495b56c48649ceacc8e1122efb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.311162 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 863ce9e1adb444b7b1399b968ea79bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.311964 2440 raft_consensus.cc:493] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:24.312101 2440 raft_consensus.cc:515] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.312855 2440 leader_election.cc:290] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:24.312958 2375 tablet_bootstrap.cc:492] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.313159 2375 ts_tablet_manager.cc:1403] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.110s user 0.005s sys 0.012s
I20251026 02:56:24.313684 2372 ts_tablet_manager.cc:1929] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.313884 2372 log.cc:1199] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7b4f977cc4cc4d078f06c4251f8c4fa0
I20251026 02:56:24.314043 2375 raft_consensus.cc:359] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.314282 2375 raft_consensus.cc:385] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.314368 2375 raft_consensus.cc:740] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.314365 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3791cb18fe1f4937badb4859d00647f3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:24.314594 2375 consensus_queue.cc:260] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.315953 2375 ts_tablet_manager.cc:1434] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 02:56:24.316296 2375 tablet_bootstrap.cc:492] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:24.318143 2195 leader_election.cc:343] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.320672 2375 tablet_bootstrap.cc:654] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.323027 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ee0a408253e4f2b942c510ae64c703e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.323525 2381 tablet_bootstrap.cc:654] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.323616 2372 ts_tablet_manager.cc:1950] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.326470 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 (table default.Table_aa762aa5113a46a98da2f4ded4e23172 [id=2b785b2f8f2d4b0fa3f24b1ea4a66cbf]) successfully deleted
I20251026 02:56:24.326876 2372 tablet_replica.cc:333] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.327112 2372 raft_consensus.cc:2243] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.327327 2372 raft_consensus.cc:2272] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.328135 2372 ts_tablet_manager.cc:1916] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.332274 2375 tablet_bootstrap.cc:492] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.333833 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 63a63b796191465aab9a63d3ffb0554d (DEFAULT_TABLE table=default.Table_b389bf6a8e5d4ca39fb2c59f8103652c [id=d27916c5ae83463e868d14a5d22ae281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.334268 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63a63b796191465aab9a63d3ffb0554d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.335873 2377 ts_tablet_manager.cc:1916] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.337036 2375 ts_tablet_manager.cc:1403] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251026 02:56:24.337832 2375 raft_consensus.cc:359] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.338016 2375 raft_consensus.cc:385] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.338102 2375 raft_consensus.cc:740] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.338305 2375 consensus_queue.cc:260] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.342813 2375 ts_tablet_manager.cc:1434] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251026 02:56:24.343940 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 63a63b796191465aab9a63d3ffb0554d (DEFAULT_TABLE table=default.Table_b389bf6a8e5d4ca39fb2c59f8103652c [id=d27916c5ae83463e868d14a5d22ae281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.344610 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94886ad6d6bb45f39261725169774ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.344681 2375 tablet_bootstrap.cc:492] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.349195 2375 tablet_bootstrap.cc:654] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.351778 2372 ts_tablet_manager.cc:1929] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.351919 2372 log.cc:1199] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/b108f0d3dae84fab93bff012400c4a2a
I20251026 02:56:24.352300 2372 ts_tablet_manager.cc:1950] T b108f0d3dae84fab93bff012400c4a2a P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.353322 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet b108f0d3dae84fab93bff012400c4a2a (table default.Table_fa379fbb05ce4e46a08b7fc1bae3183a [id=2f5d9104d21c49e0985a01ffb4b7a978]) successfully deleted
I20251026 02:56:24.353617 2393 raft_consensus.cc:493] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:24.356278 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7845ebd3e9ca4897b8fb19a48a010315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7845ebd3e9ca4897b8fb19a48a010315 already in progress: creating tablet
I20251026 02:56:24.357633 1655 catalog_manager.cc:5649] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:24.358247 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 81530bbf4a2d4e908add683475e594fb (table default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]). Expected version 1 got 0
W20251026 02:56:24.359220 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a5343ba771a4e8ea2f467c4c619c1f9 already in progress: creating tablet
W20251026 02:56:24.359506 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee already in progress: creating tablet
W20251026 02:56:24.359725 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0c647782049243d6bcd595f1a10d89c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c647782049243d6bcd595f1a10d89c6 already in progress: creating tablet
W20251026 02:56:24.359926 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0efdcc36ff5b47379e0a76512e05d379 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0efdcc36ff5b47379e0a76512e05d379 already in progress: creating tablet
I20251026 02:56:24.355159 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3791cb18fe1f4937badb4859d00647f3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:24.353832 2393 raft_consensus.cc:515] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
W20251026 02:56:24.360812 2194 leader_election.cc:343] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.361770 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2bfbe624075463f8b90ed4ec66b784c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:24.362069 2120 raft_consensus.cc:2393] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:24.364379 2393 leader_election.cc:290] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:24.365950 1962 tablet_service.cc:1505] Processing CreateTablet for tablet a08efd37968846d2958e54831488235c (DEFAULT_TABLE table=default.Table_80ca368a0ade4ac2b090b31b2ab56f12 [id=49dd7ccfa92946bebeaa4c178138f75a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.366230 2372 tablet_replica.cc:333] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.366433 2372 raft_consensus.cc:2243] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.366616 2372 raft_consensus.cc:2272] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.368816 2372 ts_tablet_manager.cc:1916] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.368760 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 352a284902c94f088e2e6ae006162db0 (DEFAULT_TABLE table=default.Table_4694119234644b988ca39446aa53f83f [id=414375003cd44b5c8c9f0249b339c28a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.369824 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 634d913c98c842a49d97a54f23752d26 (DEFAULT_TABLE table=default.Table_d2dcec579c684b92a92d98fb59ee3da1 [id=4f7edfb6d75c4a0b94ca99bb9cfe14f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.370609 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 5caeb11322fe48348ebb523967152187 (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.376394 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 0de06ea2b0544402a979d9ddc487860f (DEFAULT_TABLE table=default.Table_6546379855b749b889279973db1820cd [id=b8be7adba1cd48ae979de506d892cc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.378204 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 0fe0e06a86e241bc91a1db4219fe25bc (DEFAULT_TABLE table=default.Table_caee34582bea4cd1970c4cc59078f9d1 [id=0494cd534eb24adc94c907471cdb346a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.378826 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 863ce9e1adb444b7b1399b968ea79bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.379091 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 276bd3b97c97426daf802e1f57feb265 (DEFAULT_TABLE table=default.Table_cd142bf98f974283b7a2f9c1f1f27ef6 [id=e91358e10bab4589a6f4e815a85a5f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.379462 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 634d913c98c842a49d97a54f23752d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.380007 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a08efd37968846d2958e54831488235c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.380345 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 276bd3b97c97426daf802e1f57feb265. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.381855 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de06ea2b0544402a979d9ddc487860f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.382638 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63a63b796191465aab9a63d3ffb0554d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.382865 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe0e06a86e241bc91a1db4219fe25bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.384361 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352a284902c94f088e2e6ae006162db0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.385679 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5caeb11322fe48348ebb523967152187. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.389479 2372 ts_tablet_manager.cc:1929] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.389642 2372 log.cc:1199] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/829a6044a2da44ff90acdebca6d5b18d
I20251026 02:56:24.390072 2372 ts_tablet_manager.cc:1950] T 829a6044a2da44ff90acdebca6d5b18d P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.390164 2194 leader_election.cc:304] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:24.396483 2372 tablet_replica.cc:333] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.396709 2372 raft_consensus.cc:2243] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.396883 2372 raft_consensus.cc:2272] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:24.399739 1649 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 7439c051ca634fbe8b511d47ccede2ff (table default.Table_be7d5ddd5e4e4b15862de09dc573f7f9 [id=354a2ac9115e4f2385537f1c16992882], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7439c051ca634fbe8b511d47ccede2ff consensus state has no leader
I20251026 02:56:24.399925 2372 ts_tablet_manager.cc:1916] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.405977 2378 tablet_bootstrap.cc:492] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:24.406186 2378 ts_tablet_manager.cc:1403] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.164s user 0.008s sys 0.021s
I20251026 02:56:24.406970 2378 raft_consensus.cc:359] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.407160 2378 raft_consensus.cc:385] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.407230 2378 raft_consensus.cc:740] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.407459 2378 consensus_queue.cc:260] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.407943 2378 ts_tablet_manager.cc:1434] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:24.408363 2378 tablet_bootstrap.cc:492] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:24.410202 2378 tablet_bootstrap.cc:654] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.424757 2372 ts_tablet_manager.cc:1929] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.425069 2375 tablet_bootstrap.cc:492] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.425271 2375 ts_tablet_manager.cc:1403] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.014s
I20251026 02:56:24.426020 2375 raft_consensus.cc:359] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.426216 2375 raft_consensus.cc:385] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.426286 2375 raft_consensus.cc:740] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.426560 2375 consensus_queue.cc:260] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.427279 2375 ts_tablet_manager.cc:1434] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:24.427574 2375 tablet_bootstrap.cc:492] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.432082 2375 tablet_bootstrap.cc:654] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.432991 2372 log.cc:1199] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/c0f83d631faa468f828e54e97987a00b
I20251026 02:56:24.433395 2372 ts_tablet_manager.cc:1950] T c0f83d631faa468f828e54e97987a00b P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.435453 2372 tablet_replica.cc:333] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.435633 2372 raft_consensus.cc:2243] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.435798 2372 raft_consensus.cc:2272] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.437978 2372 ts_tablet_manager.cc:1916] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.442011 2375 tablet_bootstrap.cc:492] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.442168 2375 ts_tablet_manager.cc:1403] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20251026 02:56:24.443003 2375 raft_consensus.cc:359] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.443182 2375 raft_consensus.cc:385] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.443284 2375 raft_consensus.cc:740] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.449095 2375 consensus_queue.cc:260] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.449714 2375 ts_tablet_manager.cc:1434] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251026 02:56:24.456604 2375 tablet_bootstrap.cc:492] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.458289 2375 tablet_bootstrap.cc:654] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.463270 2375 tablet_bootstrap.cc:492] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.463469 2375 ts_tablet_manager.cc:1403] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20251026 02:56:24.463763 2372 ts_tablet_manager.cc:1929] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.463891 2372 log.cc:1199] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/b435520cadb344edaa29975b4e19718a
I20251026 02:56:24.464489 2372 ts_tablet_manager.cc:1950] T b435520cadb344edaa29975b4e19718a P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.465776 2372 tablet_replica.cc:333] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.466017 2372 raft_consensus.cc:2243] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.465917 2375 raft_consensus.cc:359] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.466166 2375 raft_consensus.cc:385] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.466187 2372 raft_consensus.cc:2272] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.466259 2375 raft_consensus.cc:740] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.466512 2375 consensus_queue.cc:260] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.467065 2372 ts_tablet_manager.cc:1916] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:24.468262 2485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20251026 02:56:24.468958 2375 ts_tablet_manager.cc:1434] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20251026 02:56:24.469126 2485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20251026 02:56:24.469278 2375 tablet_bootstrap.cc:492] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:24.469400 2485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251026 02:56:24.476378 2237 tablet_service.cc:1505] Processing CreateTablet for tablet a08efd37968846d2958e54831488235c (DEFAULT_TABLE table=default.Table_80ca368a0ade4ac2b090b31b2ab56f12 [id=49dd7ccfa92946bebeaa4c178138f75a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.478230 2376 ts_tablet_manager.cc:1929] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.478381 2376 log.cc:1199] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/7b4f977cc4cc4d078f06c4251f8c4fa0
I20251026 02:56:24.478420 2375 tablet_bootstrap.cc:654] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.478761 2376 ts_tablet_manager.cc:1950] T 7b4f977cc4cc4d078f06c4251f8c4fa0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:24.391209 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 829a6044a2da44ff90acdebca6d5b18d (table default.Table_6546379855b749b889279973db1820cd [id=b8be7adba1cd48ae979de506d892cc3d]) successfully deleted
W20251026 02:56:24.481783 1648 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 90603.4us
I20251026 02:56:24.482226 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet c0f83d631faa468f828e54e97987a00b (table default.Table_4578a4ce81ea42cd878859a8167babff [id=f914c9974ac44d3f84c3d312649fe255]) successfully deleted
I20251026 02:56:24.482422 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet b435520cadb344edaa29975b4e19718a (table default.Table_66dd770512e847f4a14ee04e5afa3a6c [id=72fa811ef7224d42a83e602c3707f028]) successfully deleted
I20251026 02:56:24.483294 2096 tablet_service.cc:1505] Processing CreateTablet for tablet f7da859119bd42ea9fa1290b3d4bf88c (DEFAULT_TABLE table=default.Table_b22d8322842a4ff6ad61fc3a8ad64f62 [id=5333d696902241b89eadf7e9e627c41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.484282 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7da859119bd42ea9fa1290b3d4bf88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.484436 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 2487bf7d40f34922aa2fd165488342f6 with delete_type TABLET_DATA_DELETED (replaced by tablet a111848706b743c9beb2fbd21f59d7f3) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.485076 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e89c69e3b58458aad10beaeb5128d60 with delete_type TABLET_DATA_DELETED (replaced by tablet a821ed8a686c452a975acb96b5c74778) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:24.485764 2485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 02:56:24.485860 2485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 02:56:24.485937 2485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 02:56:24.486003 2485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 02:56:24.487291 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 829a6044a2da44ff90acdebca6d5b18d with delete_type TABLET_DATA_DELETED (replaced by tablet 0de06ea2b0544402a979d9ddc487860f) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.495013 2405 raft_consensus.cc:2749] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:24.495426 2372 ts_tablet_manager.cc:1929] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.495769 2372 log.cc:1199] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/565255802f4f42ba8320a3ec80d2db8f
I20251026 02:56:24.496276 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a08efd37968846d2958e54831488235c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.496292 2372 ts_tablet_manager.cc:1950] T 565255802f4f42ba8320a3ec80d2db8f P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.497117 2375 tablet_bootstrap.cc:492] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.497319 2375 ts_tablet_manager.cc:1403] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20251026 02:56:24.497530 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2bfbe624075463f8b90ed4ec66b784c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:24.498450 2375 raft_consensus.cc:359] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.498704 2375 raft_consensus.cc:385] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.498819 2375 raft_consensus.cc:740] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.499172 2375 consensus_queue.cc:260] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.502714 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 634d913c98c842a49d97a54f23752d26 (DEFAULT_TABLE table=default.Table_d2dcec579c684b92a92d98fb59ee3da1 [id=4f7edfb6d75c4a0b94ca99bb9cfe14f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.503990 2375 ts_tablet_manager.cc:1434] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251026 02:56:24.504640 2375 tablet_bootstrap.cc:492] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.507421 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet ad55040f0a2b4cfa894d1bd058d4d69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.508351 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 0de06ea2b0544402a979d9ddc487860f (DEFAULT_TABLE table=default.Table_6546379855b749b889279973db1820cd [id=b8be7adba1cd48ae979de506d892cc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.509214 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de06ea2b0544402a979d9ddc487860f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.509176 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 7036a159e88b4171942dd238f8af3ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.509931 2381 tablet_bootstrap.cc:492] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:24.510111 2381 ts_tablet_manager.cc:1403] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.228s user 0.001s sys 0.005s
I20251026 02:56:24.510489 2405 raft_consensus.cc:493] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:24.510660 2405 raft_consensus.cc:515] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.511349 2405 leader_election.cc:290] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:24.511451 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 565255802f4f42ba8320a3ec80d2db8f (table default.Table_cf4652bce4e447caa066bfebc7b43e50 [id=1f46f0e30c4045419969ccc51df359f1]) successfully deleted
I20251026 02:56:24.512058 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c1596dd4394419a8bb6ea6a84be9fea" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:24.512619 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 352a284902c94f088e2e6ae006162db0 (DEFAULT_TABLE table=default.Table_4694119234644b988ca39446aa53f83f [id=414375003cd44b5c8c9f0249b339c28a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.513408 2375 tablet_bootstrap.cc:654] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.514379 2485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.150.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.150.60:33729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.150.60
--webserver_port=39671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 26 Oct 2025 02:43:21 UTC on 24a791456cd2
build id 8720
ASAN enabled
I20251026 02:56:24.514724 2372 tablet_replica.cc:333] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.514986 2372 raft_consensus.cc:2243] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.515232 2372 raft_consensus.cc:2272] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.515653 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352a284902c94f088e2e6ae006162db0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.516149 2372 ts_tablet_manager.cc:1916] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.513357 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 5caeb11322fe48348ebb523967152187 (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.518267 1915 leader_election.cc:304] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:24.518767 2393 raft_consensus.cc:3060] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:24.521103 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3316a969a3654fdf853bca2d3d666aa4" } new_table_name: "default.Table_241789d6111d497c89efdb4507dff32e" modify_external_catalogs: true
I20251026 02:56:24.521793 2393 raft_consensus.cc:2749] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:24.522362 2194 leader_election.cc:343] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.522542 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_caee34582bea4cd1970c4cc59078f9d1" } modify_external_catalogs: true
I20251026 02:56:24.522787 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_caee34582bea4cd1970c4cc59078f9d1" } modify_external_catalogs: true
I20251026 02:56:24.524340 2381 raft_consensus.cc:359] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.524650 2381 raft_consensus.cc:385] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.524875 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c1596dd4394419a8bb6ea6a84be9fea" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:24.528185 2381 raft_consensus.cc:740] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.528641 2381 consensus_queue.cc:260] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.531966 2485 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 02:56:24.535660 2381 ts_tablet_manager.cc:1434] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.020s user 0.000s sys 0.003s
W20251026 02:56:24.536839 2195 leader_election.cc:343] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.537103 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 634d913c98c842a49d97a54f23752d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.537150 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 276bd3b97c97426daf802e1f57feb265 (DEFAULT_TABLE table=default.Table_cd142bf98f974283b7a2f9c1f1f27ef6 [id=e91358e10bab4589a6f4e815a85a5f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.537461 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5caeb11322fe48348ebb523967152187. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.537554 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 0fe0e06a86e241bc91a1db4219fe25bc (DEFAULT_TABLE table=default.Table_caee34582bea4cd1970c4cc59078f9d1 [id=0494cd534eb24adc94c907471cdb346a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.537905 2195 leader_election.cc:304] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:24.538394 2405 raft_consensus.cc:2749] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:24.539177 2381 tablet_bootstrap.cc:492] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:24.540854 2485 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 02:56:24.544521 2381 tablet_bootstrap.cc:654] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.544541 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0fe0e06a86e241bc91a1db4219fe25bc
I20251026 02:56:24.544818 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 276bd3b97c97426daf802e1f57feb265. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.545336 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe0e06a86e241bc91a1db4219fe25bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.545766 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 0fe0e06a86e241bc91a1db4219fe25bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:24.546540 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ced7ad6a601d432cbc13b3199a0506e4 consensus state has no leader
I20251026 02:56:24.548588 2375 tablet_bootstrap.cc:492] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.548836 2375 ts_tablet_manager.cc:1403] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20251026 02:56:24.549916 2375 raft_consensus.cc:359] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.550197 2375 raft_consensus.cc:385] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.550302 2375 raft_consensus.cc:740] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.550594 2375 consensus_queue.cc:260] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.554225 2375 ts_tablet_manager.cc:1434] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251026 02:56:24.554912 2375 tablet_bootstrap.cc:492] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.562319 2372 ts_tablet_manager.cc:1929] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.562521 2372 log.cc:1199] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/2ee0a408253e4f2b942c510ae64c703e
I20251026 02:56:24.562984 2372 ts_tablet_manager.cc:1950] T 2ee0a408253e4f2b942c510ae64c703e P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.564320 2375 tablet_bootstrap.cc:654] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.567346 2372 tablet_replica.cc:333] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.567585 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 2ee0a408253e4f2b942c510ae64c703e (table default.Table_c550a959e98a4e22853deca526de3eae [id=47bcd084e7fa4dec9f23cef12d10aa46]) successfully deleted
I20251026 02:56:24.567674 2372 raft_consensus.cc:2243] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.567896 2372 raft_consensus.cc:2272] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:24.567989 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 829a6044a2da44ff90acdebca6d5b18d because the tablet was not found. No further retry: Not found: Tablet not found: 829a6044a2da44ff90acdebca6d5b18d
I20251026 02:56:24.568881 2372 ts_tablet_manager.cc:1916] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:24.576486 2604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 02:56:24.576889 2375 tablet_bootstrap.cc:492] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.577589 2375 ts_tablet_manager.cc:1403] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.007s
I20251026 02:56:24.578800 2375 raft_consensus.cc:359] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.579066 2375 raft_consensus.cc:385] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.579195 2375 raft_consensus.cc:740] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.579488 2375 consensus_queue.cc:260] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.580471 2375 ts_tablet_manager.cc:1434] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20251026 02:56:24.583994 2605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 02:56:24.583464 2375 tablet_bootstrap.cc:492] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.590796 2377 ts_tablet_manager.cc:1929] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.590976 2377 log.cc:1199] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/829a6044a2da44ff90acdebca6d5b18d
I20251026 02:56:24.591374 2377 ts_tablet_manager.cc:1950] T 829a6044a2da44ff90acdebca6d5b18d P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:24.594120 2372 ts_tablet_manager.cc:1929] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.594357 2372 log.cc:1199] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/ad55040f0a2b4cfa894d1bd058d4d69f
I20251026 02:56:24.594825 2372 ts_tablet_manager.cc:1950] T ad55040f0a2b4cfa894d1bd058d4d69f P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.596400 2485 server_base.cc:1047] running on GCE node
I20251026 02:56:24.597074 2375 tablet_bootstrap.cc:654] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:24.598590 2608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 02:56:24.599869 2485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 02:56:24.600404 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet ad55040f0a2b4cfa894d1bd058d4d69f (table default.Table_0da51595453549b3b7e077f7b7ff3ec7 [id=53cd517e68e64597ad51d85d16f0399d]) successfully deleted
I20251026 02:56:24.600934 2372 tablet_replica.cc:333] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.601222 2372 raft_consensus.cc:2243] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.601503 2372 raft_consensus.cc:2272] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.602855 2375 tablet_bootstrap.cc:492] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.603081 2375 ts_tablet_manager.cc:1403] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s
I20251026 02:56:24.604157 2375 raft_consensus.cc:359] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.604470 2375 raft_consensus.cc:385] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.604568 2375 raft_consensus.cc:740] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.604916 2375 consensus_queue.cc:260] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.606108 2375 ts_tablet_manager.cc:1434] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251026 02:56:24.608985 2375 tablet_bootstrap.cc:492] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.610345 2485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 02:56:24.612375 2485 hybrid_clock.cc:648] HybridClock initialized: now 1761447384612320 us; error 36 us; skew 500 ppm
I20251026 02:56:24.612800 2372 ts_tablet_manager.cc:1916] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.612860 2485 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 02:56:24.616372 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet fd1484434dd64f4bb2764268b3d3b218 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.627561 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c1596dd4394419a8bb6ea6a84be9fea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.628780 2485 webserver.cc:492] Webserver started at http://127.1.150.60:39671/ using document root <none> and password file <none>
I20251026 02:56:24.630183 2375 tablet_bootstrap.cc:654] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.631625 2485 fs_manager.cc:362] Metadata directory not provided
I20251026 02:56:24.632311 2485 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 02:56:24.632858 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7be811f036924663898ad10ce19955b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:24.633638 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7be811f036924663898ad10ce19955b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:24.634423 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_537013d780884ee89ee2ad679f506a49" } new_table_name: "default.Table_9f3815b1bf464a83a574e3075d065956" modify_external_catalogs: true
I20251026 02:56:24.634366 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d48e174e391f42d6aa0b05c322b5b828"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:24.635171 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_87f51d8912864f0aa7ef8b490392611e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:24.635740 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87f51d8912864f0aa7ef8b490392611e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:24.636444 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d48e174e391f42d6aa0b05c322b5b828 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:24.637145 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7b6687267a1a4b2da7e02949e7b4c081" } modify_external_catalogs: true
I20251026 02:56:24.637193 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f2809b812fe14b2d9e44b22bfa68c1f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:24.637413 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7b6687267a1a4b2da7e02949e7b4c081" } modify_external_catalogs: true
I20251026 02:56:24.637476 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cdb10d198a394aac9033a98152fd2d94" } modify_external_catalogs: true
I20251026 02:56:24.637641 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cdb10d198a394aac9033a98152fd2d94" } modify_external_catalogs: true
W20251026 02:56:24.637729 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2809b812fe14b2d9e44b22bfa68c1f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:24.637850 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7b08384413864da0a28224dd7201c263"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:24.638522 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b08384413864da0a28224dd7201c263 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:24.639003 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:24.639645 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet b108f0d3dae84fab93bff012400c4a2a (table default.Table_fa379fbb05ce4e46a08b7fc1bae3183a [id=2f5d9104d21c49e0985a01ffb4b7a978]) successfully deleted
W20251026 02:56:24.639724 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:24.640051 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a5343ba771a4e8ea2f467c4c619c1f9 already in progress: creating tablet
I20251026 02:56:24.641443 2378 tablet_bootstrap.cc:492] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:24.641647 2378 ts_tablet_manager.cc:1403] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.233s user 0.045s sys 0.128s
I20251026 02:56:24.642650 2378 raft_consensus.cc:359] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.642827 2378 raft_consensus.cc:385] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.642910 2378 raft_consensus.cc:740] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.643191 2378 consensus_queue.cc:260] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.643741 2372 ts_tablet_manager.cc:1929] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.643811 2378 ts_tablet_manager.cc:1434] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:24.643877 2372 log.cc:1199] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7036a159e88b4171942dd238f8af3ed3
I20251026 02:56:24.644120 2378 tablet_bootstrap.cc:492] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:24.644626 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_09ca38422af543f89eab26b4117536dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:24.645501 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09ca38422af543f89eab26b4117536dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:24.645761 2376 tablet_replica.cc:333] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:24.645953 2376 raft_consensus.cc:2243] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.646116 2376 raft_consensus.cc:2272] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.646678 2376 ts_tablet_manager.cc:1916] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.650590 2375 tablet_bootstrap.cc:492] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.650760 2375 ts_tablet_manager.cc:1403] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.002s
I20251026 02:56:24.651561 2375 raft_consensus.cc:359] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.651737 2375 raft_consensus.cc:385] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.651822 2375 raft_consensus.cc:740] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.652055 2375 consensus_queue.cc:260] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:24.652510 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0efdcc36ff5b47379e0a76512e05d379 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0efdcc36ff5b47379e0a76512e05d379 already in progress: creating tablet
I20251026 02:56:24.652577 2375 ts_tablet_manager.cc:1434] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251026 02:56:24.652810 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee already in progress: creating tablet
W20251026 02:56:24.653033 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0c647782049243d6bcd595f1a10d89c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c647782049243d6bcd595f1a10d89c6 already in progress: creating tablet
I20251026 02:56:24.653748 2375 tablet_bootstrap.cc:492] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.655519 2375 tablet_bootstrap.cc:654] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.656258 2372 ts_tablet_manager.cc:1950] T 7036a159e88b4171942dd238f8af3ed3 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.657382 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7036a159e88b4171942dd238f8af3ed3 (table default.Table_7e5c0b5970974e12aba66398ec59430e [id=59edc20bb9be438c9ef00aafcd1c6376]) successfully deleted
I20251026 02:56:24.660950 2372 tablet_replica.cc:333] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.661188 2372 raft_consensus.cc:2243] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.661355 2372 raft_consensus.cc:2272] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:24.663012 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0fe0e06a86e241bc91a1db4219fe25bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe0e06a86e241bc91a1db4219fe25bc already in progress: creating tablet
I20251026 02:56:24.663712 2372 ts_tablet_manager.cc:1916] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:24.664434 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 829a6044a2da44ff90acdebca6d5b18d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829a6044a2da44ff90acdebca6d5b18d already in progress: creating tablet
I20251026 02:56:24.669788 2378 tablet_bootstrap.cc:654] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.670043 2485 fs_manager.cc:714] Time spent opening directory manager: real 0.030s user 0.010s sys 0.000s
I20251026 02:56:24.671389 2381 tablet_bootstrap.cc:492] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:24.671562 2381 ts_tablet_manager.cc:1403] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.132s user 0.018s sys 0.041s
I20251026 02:56:24.672827 2376 ts_tablet_manager.cc:1929] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.672961 2376 log.cc:1199] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/565255802f4f42ba8320a3ec80d2db8f
I20251026 02:56:24.673273 2376 ts_tablet_manager.cc:1950] T 565255802f4f42ba8320a3ec80d2db8f P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:24.674799 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 7b4f977cc4cc4d078f06c4251f8c4fa0 (table default.Table_aa762aa5113a46a98da2f4ded4e23172 [id=2b785b2f8f2d4b0fa3f24b1ea4a66cbf]) successfully deleted
I20251026 02:56:24.674878 2376 tablet_replica.cc:333] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:24.675041 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 565255802f4f42ba8320a3ec80d2db8f (table default.Table_cf4652bce4e447caa066bfebc7b43e50 [id=1f46f0e30c4045419969ccc51df359f1]) successfully deleted
I20251026 02:56:24.675055 2376 raft_consensus.cc:2243] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.675246 2376 raft_consensus.cc:2272] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.675724 2381 raft_consensus.cc:359] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.675944 2381 raft_consensus.cc:385] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.676023 2381 raft_consensus.cc:740] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.675969 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 2524ee0c964345459c3e62f1e39b8f9e (DEFAULT_TABLE table=default.Table_6ce7e6a26747479bb5b4a5b04d37105a [id=91b4b1a8162441299c680e7c202a96d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.676373 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2524ee0c964345459c3e62f1e39b8f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.676290 2381 consensus_queue.cc:260] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.676808 2381 ts_tablet_manager.cc:1434] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251026 02:56:24.681173 1953 tablet_service.cc:1505] Processing CreateTablet for tablet f7da859119bd42ea9fa1290b3d4bf88c (DEFAULT_TABLE table=default.Table_b22d8322842a4ff6ad61fc3a8ad64f62 [id=5333d696902241b89eadf7e9e627c41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.681337 2381 tablet_bootstrap.cc:492] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:24.681560 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7da859119bd42ea9fa1290b3d4bf88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.682969 2381 tablet_bootstrap.cc:654] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.684376 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet b435520cadb344edaa29975b4e19718a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.685828 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_34881d9847d5463297830bf6cc4ed1e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:24.686337 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0ea3c4e29f264b34b9f6030b92262af6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:24.686964 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ea3c4e29f264b34b9f6030b92262af6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:24.688304 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 2524ee0c964345459c3e62f1e39b8f9e (DEFAULT_TABLE table=default.Table_6ce7e6a26747479bb5b4a5b04d37105a [id=91b4b1a8162441299c680e7c202a96d9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:24.688550 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34881d9847d5463297830bf6cc4ed1e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:24.688743 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2524ee0c964345459c3e62f1e39b8f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.693054 2240 tablet_service.cc:1505] Processing CreateTablet for tablet f7da859119bd42ea9fa1290b3d4bf88c (DEFAULT_TABLE table=default.Table_b22d8322842a4ff6ad61fc3a8ad64f62 [id=5333d696902241b89eadf7e9e627c41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.693413 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7da859119bd42ea9fa1290b3d4bf88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.694229 2393 raft_consensus.cc:493] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:24.694453 2393 raft_consensus.cc:515] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } }
I20251026 02:56:24.695189 2393 leader_election.cc:290] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:24.696404 1916 leader_election.cc:343] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 0e89c69e3b58458aad10beaeb5128d60
W20251026 02:56:24.696946 1915 leader_election.cc:343] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 0e89c69e3b58458aad10beaeb5128d60
I20251026 02:56:24.697013 2372 ts_tablet_manager.cc:1929] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.697067 1915 leader_election.cc:304] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:24.697155 2372 log.cc:1199] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/fd1484434dd64f4bb2764268b3d3b218
I20251026 02:56:24.697525 2372 ts_tablet_manager.cc:1950] T fd1484434dd64f4bb2764268b3d3b218 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.697569 2393 raft_consensus.cc:2749] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:24.704490 2372 tablet_replica.cc:333] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.704737 2372 raft_consensus.cc:2243] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.704871 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet fd1484434dd64f4bb2764268b3d3b218 (table default.Table_f43d7cfe30cd40c08a88e4a62da5cbf4 [id=d342cbe7a7cb44928ec73415b30a779e]) successfully deleted
I20251026 02:56:24.704985 2372 raft_consensus.cc:2272] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.706130 2372 ts_tablet_manager.cc:1916] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.712783 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8dbc12d49b094dbeadad2e76916ba9af
I20251026 02:56:24.714491 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 8dbc12d49b094dbeadad2e76916ba9af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.714910 2619 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:24.716682 2375 tablet_bootstrap.cc:492] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.716856 2375 ts_tablet_manager.cc:1403] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.007s
I20251026 02:56:24.718021 2375 raft_consensus.cc:359] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.718261 2375 raft_consensus.cc:385] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.718351 2375 raft_consensus.cc:740] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.718624 2375 consensus_queue.cc:260] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.720049 2375 ts_tablet_manager.cc:1434] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:24.720429 2375 tablet_bootstrap.cc:492] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.722206 2375 tablet_bootstrap.cc:654] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.724473 2485 fs_manager.cc:730] Time spent opening block manager: real 0.052s user 0.011s sys 0.004s
I20251026 02:56:24.725337 2372 ts_tablet_manager.cc:1929] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.725481 2372 log.cc:1199] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/1c1596dd4394419a8bb6ea6a84be9fea
I20251026 02:56:24.724680 2485 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data,/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
uuid: "751b719b75ce4bc2a88a19091230ecb7"
format_stamp: "Formatted at 2025-10-26 02:56:18 on dist-test-slave-h3w0"
I20251026 02:56:24.725881 2372 ts_tablet_manager.cc:1950] T 1c1596dd4394419a8bb6ea6a84be9fea P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.725692 2485 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 02:56:24.726939 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 1c1596dd4394419a8bb6ea6a84be9fea (table default.Table_4a7c7fde2faf424590cdf3e2ddf0c1a5 [id=d9c0bfe3ef1145189ccef147212f787b]) successfully deleted
W20251026 02:56:24.728770 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 8dbc12d49b094dbeadad2e76916ba9af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dbc12d49b094dbeadad2e76916ba9af already in progress: creating tablet
I20251026 02:56:24.730522 2376 ts_tablet_manager.cc:1916] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.734817 2378 tablet_bootstrap.cc:492] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:24.735034 2378 ts_tablet_manager.cc:1403] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.091s user 0.010s sys 0.001s
I20251026 02:56:24.736025 2378 raft_consensus.cc:359] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.739245 2375 tablet_bootstrap.cc:492] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.739463 2375 ts_tablet_manager.cc:1403] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20251026 02:56:24.739702 2378 raft_consensus.cc:385] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.739821 2378 raft_consensus.cc:740] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.740110 2378 consensus_queue.cc:260] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.741397 2378 ts_tablet_manager.cc:1434] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251026 02:56:24.741832 2375 raft_consensus.cc:359] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.742072 2375 raft_consensus.cc:385] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.742199 2375 raft_consensus.cc:740] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.742472 2375 consensus_queue.cc:260] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.743907 2120 raft_consensus.cc:1275] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:24.744607 2405 consensus_queue.cc:1048] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:24.745504 2375 ts_tablet_manager.cc:1434] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251026 02:56:24.747268 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 115ccccf16b74efc9f24b607cf0c74d1
I20251026 02:56:24.747977 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet 565255802f4f42ba8320a3ec80d2db8f from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:24.748498 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet 20dec74440cf4bb9949c462ec51e9889 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:24.750823 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_133a36cbb31242b680078785eb21866e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:24.751510 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_133a36cbb31242b680078785eb21866e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:24.752017 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_25607539f64440cb8ff2c24a0a3a3bdd" } modify_external_catalogs: true
I20251026 02:56:24.752334 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 115ccccf16b74efc9f24b607cf0c74d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.752869 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cfeac8daaf08492a9456d2e06cd62a58" } new_table_name: "default.Table_389946812a3d4337b46c0b2829ab935f" modify_external_catalogs: true
I20251026 02:56:24.753644 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_25607539f64440cb8ff2c24a0a3a3bdd" } modify_external_catalogs: true
I20251026 02:56:24.754434 2378 tablet_bootstrap.cc:492] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:24.755972 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet 7bc92f90a68b4e749bfd4fa49076bfe2 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:24.757702 2194 consensus_peers.cc:597] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:24.757838 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e70cf6d4e0dc4017b484753f4b0250d9" } modify_external_catalogs: true
I20251026 02:56:24.758033 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e70cf6d4e0dc4017b484753f4b0250d9" } modify_external_catalogs: true
I20251026 02:56:24.758646 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cd142bf98f974283b7a2f9c1f1f27ef6" } new_table_name: "default.Table_6e87e5c48f4a4aa8be0b0ecc837e20dd" modify_external_catalogs: true
I20251026 02:56:24.759049 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet ad55040f0a2b4cfa894d1bd058d4d69f from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:24.759470 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet cfed3523205548508d1101586ee3e2a9 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:24.759979 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4694119234644b988ca39446aa53f83f" } new_table_name: "default.Table_6cd4ee82e9ec406f971669b7f0722677" modify_external_catalogs: true
W20251026 02:56:24.761250 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 115ccccf16b74efc9f24b607cf0c74d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 115ccccf16b74efc9f24b607cf0c74d1 already in progress: creating tablet
I20251026 02:56:24.761479 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.763890 2372 tablet_replica.cc:333] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.766464 2372 raft_consensus.cc:2243] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.766769 2372 raft_consensus.cc:2272] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.767792 2372 ts_tablet_manager.cc:1916] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.767829 2506 raft_consensus.cc:493] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:24.768132 2506 raft_consensus.cc:515] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.768998 2506 leader_election.cc:290] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:24.769984 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d8cdf79a97149618bcaf0fefee1dfeb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:24.771453 2378 tablet_bootstrap.cc:654] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.771631 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d8cdf79a97149618bcaf0fefee1dfeb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:24.772253 2053 leader_election.cc:343] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:24.772526 2053 leader_election.cc:343] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.772636 2053 leader_election.cc:304] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:24.772903 2375 tablet_bootstrap.cc:492] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.773056 2506 raft_consensus.cc:2749] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:24.775166 2629 tablet.cc:1722] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:24.775462 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c1596dd4394419a8bb6ea6a84be9fea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.775717 2377 tablet_replica.cc:333] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:24.775964 2377 raft_consensus.cc:2243] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.776139 2377 raft_consensus.cc:2272] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.776113 2376 ts_tablet_manager.cc:1929] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.776263 2376 log.cc:1199] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/2487bf7d40f34922aa2fd165488342f6
I20251026 02:56:24.776751 2377 ts_tablet_manager.cc:1916] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:24.781530 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 740f7c7f186f45ceaa4c40e76419e3a1 consensus state has no leader
I20251026 02:56:24.782371 2376 ts_tablet_manager.cc:1950] T 2487bf7d40f34922aa2fd165488342f6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:24.783527 2628 tablet.cc:1722] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:24.783671 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 81530bbf4a2d4e908add683475e594fb (table default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]). Expected version 1 got 0
I20251026 02:56:24.784036 2375 tablet_bootstrap.cc:654] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.785244 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d6690e6bf67454398637d9908b9c652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.786223 2376 tablet_replica.cc:333] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:24.786500 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7bc92f90a68b4e749bfd4fa49076bfe2
I20251026 02:56:24.787029 1656 catalog_manager.cc:6719] ReplaceTablet: tablet 7bc92f90a68b4e749bfd4fa49076bfe2 deleted and replaced by tablet 17d2cc6b16d34331bb80c576c0cd2ce0
I20251026 02:56:24.787639 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 17d2cc6b16d34331bb80c576c0cd2ce0) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.787868 2372 ts_tablet_manager.cc:1929] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.788014 2372 log.cc:1199] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3f60bcc5e4cb4fd19d2a5713f69814cd
I20251026 02:56:24.788497 2372 ts_tablet_manager.cc:1950] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.789655 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3f60bcc5e4cb4fd19d2a5713f69814cd (table default.Table_d029698b06db4a6f807451acb73a3680 [id=2ae19615aa104e2b8791b86755e1184f]) successfully deleted
I20251026 02:56:24.790766 2376 raft_consensus.cc:2243] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.791623 2376 raft_consensus.cc:2272] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.791895 2405 raft_consensus.cc:493] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:24.791963 2372 tablet_replica.cc:333] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.792068 2405 raft_consensus.cc:515] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.792184 2372 raft_consensus.cc:2243] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.792742 2405 leader_election.cc:290] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:24.793172 2375 tablet_bootstrap.cc:492] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.793404 2372 raft_consensus.cc:2272] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.793645 2375 ts_tablet_manager.cc:1403] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20251026 02:56:24.793627 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd1484434dd64f4bb2764268b3d3b218" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:24.794037 2376 ts_tablet_manager.cc:1916] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.794225 2372 ts_tablet_manager.cc:1916] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:24.794252 2194 leader_election.cc:343] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.794499 2375 raft_consensus.cc:359] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.794727 2375 raft_consensus.cc:385] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.794816 2375 raft_consensus.cc:740] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.795089 2375 consensus_queue.cc:260] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.795735 2375 ts_tablet_manager.cc:1434] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251026 02:56:24.795729 2195 leader_election.cc:343] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: fd1484434dd64f4bb2764268b3d3b218
I20251026 02:56:24.795864 2195 leader_election.cc:304] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:24.796128 2375 tablet_bootstrap.cc:492] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.796394 2405 raft_consensus.cc:2749] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:24.800758 2375 tablet_bootstrap.cc:654] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.804838 2372 ts_tablet_manager.cc:1929] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.804996 2372 log.cc:1199] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3d6690e6bf67454398637d9908b9c652
I20251026 02:56:24.805351 2372 ts_tablet_manager.cc:1950] T 3d6690e6bf67454398637d9908b9c652 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.806401 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3d6690e6bf67454398637d9908b9c652 (table default.Table_e071f7a8a409464d810f12e647a4ff0e [id=c98c6fb4910449b0ad4513fc1c8f6f36]) successfully deleted
I20251026 02:56:24.806607 2375 tablet_bootstrap.cc:492] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.806773 2375 ts_tablet_manager.cc:1403] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20251026 02:56:24.806905 2372 tablet_replica.cc:333] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.807248 2372 raft_consensus.cc:2243] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.807493 2372 raft_consensus.cc:2272] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.808036 2372 ts_tablet_manager.cc:1916] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:24.808810 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 352a284902c94f088e2e6ae006162db0 consensus state has no leader
W20251026 02:56:24.809283 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 276bd3b97c97426daf802e1f57feb265 consensus state has no leader
I20251026 02:56:24.811870 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (DEFAULT_TABLE table=default.Table_7be811f036924663898ad10ce19955b1 [id=44c37945ff9f48e8b2c07648394ac177]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.812345 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.815114 2372 ts_tablet_manager.cc:1929] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:24.815186 2377 ts_tablet_manager.cc:1929] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.815243 2372 log.cc:1199] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7bc92f90a68b4e749bfd4fa49076bfe2
I20251026 02:56:24.815332 2377 log.cc:1199] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/1c1596dd4394419a8bb6ea6a84be9fea
I20251026 02:56:24.815685 2372 ts_tablet_manager.cc:1950] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.815692 2377 ts_tablet_manager.cc:1950] T 1c1596dd4394419a8bb6ea6a84be9fea P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:24.807626 2375 raft_consensus.cc:359] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.815830 2375 raft_consensus.cc:385] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.815943 2375 raft_consensus.cc:740] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.816279 2375 consensus_queue.cc:260] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.816815 2506 raft_consensus.cc:493] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:24.817000 2375 ts_tablet_manager.cc:1434] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251026 02:56:24.817035 2506 raft_consensus.cc:515] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.817364 2375 tablet_bootstrap.cc:492] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:24.817822 2506 leader_election.cc:290] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:24.818316 2381 tablet_bootstrap.cc:492] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:24.818648 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8ae08997af74b66b7d6375212eee4c2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:24.820245 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8ae08997af74b66b7d6375212eee4c2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:24.822764 2053 leader_election.cc:343] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:24.823120 2053 leader_election.cc:343] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.823343 2053 leader_election.cc:304] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:24.823697 2506 raft_consensus.cc:2749] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:24.824537 2376 ts_tablet_manager.cc:1929] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.824703 2376 log.cc:1199] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/0e89c69e3b58458aad10beaeb5128d60
I20251026 02:56:24.825057 2376 ts_tablet_manager.cc:1950] T 0e89c69e3b58458aad10beaeb5128d60 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:24.825994 2376 tablet_replica.cc:333] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:24.826206 2376 raft_consensus.cc:2243] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.826391 2376 raft_consensus.cc:2272] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.828747 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1e56b259a314422aba35eb129021bbeb
I20251026 02:56:24.828827 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (table default.Table_9f3815b1bf464a83a574e3075d065956 [id=095ad208529348d4aea2d4e2840e07c7]) successfully deleted
I20251026 02:56:24.829600 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet cfed3523205548508d1101586ee3e2a9
I20251026 02:56:24.829708 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet f8533f1c191042ee9f2c0b99972fe931
I20251026 02:56:24.830025 1657 catalog_manager.cc:6719] ReplaceTablet: tablet cfed3523205548508d1101586ee3e2a9 deleted and replaced by tablet 44af74033ccf4d048475f12d164938f5
I20251026 02:56:24.830649 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet f8533f1c191042ee9f2c0b99972fe931 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.830847 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet cfed3523205548508d1101586ee3e2a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 44af74033ccf4d048475f12d164938f5) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.831466 2375 tablet_bootstrap.cc:654] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.833192 1656 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8] alter complete (version 1)
I20251026 02:56:24.833814 2376 ts_tablet_manager.cc:1916] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.836931 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet 0fe0e06a86e241bc91a1db4219fe25bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.836967 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 7036a159e88b4171942dd238f8af3ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.837379 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 0fe0e06a86e241bc91a1db4219fe25bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.838665 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e56b259a314422aba35eb129021bbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.840251 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 829a6044a2da44ff90acdebca6d5b18d (table default.Table_6546379855b749b889279973db1820cd [id=b8be7adba1cd48ae979de506d892cc3d]) successfully deleted
W20251026 02:56:24.842003 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet f8533f1c191042ee9f2c0b99972fe931 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8533f1c191042ee9f2c0b99972fe931 already in progress: creating tablet
I20251026 02:56:24.842621 2377 tablet_replica.cc:333] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:24.842969 2377 raft_consensus.cc:2243] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.843267 2377 raft_consensus.cc:2272] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.844353 2377 ts_tablet_manager.cc:1916] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:24.845125 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet cfed3523205548508d1101586ee3e2a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfed3523205548508d1101586ee3e2a9 already in progress: creating tablet
W20251026 02:56:24.845958 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 1e56b259a314422aba35eb129021bbeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e56b259a314422aba35eb129021bbeb already in progress: creating tablet
I20251026 02:56:24.846983 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet 829a6044a2da44ff90acdebca6d5b18d with delete_type TABLET_DATA_DELETED (replaced by tablet 0de06ea2b0544402a979d9ddc487860f) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.848125 2381 ts_tablet_manager.cc:1403] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.167s user 0.002s sys 0.005s
I20251026 02:56:24.848315 2506 raft_consensus.cc:493] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:24.848555 2506 raft_consensus.cc:515] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.849205 2381 raft_consensus.cc:359] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.849325 2506 leader_election.cc:290] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:24.849428 2381 raft_consensus.cc:385] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.849537 2381 raft_consensus.cc:740] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.849877 2381 consensus_queue.cc:260] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.850672 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bcd7dec9c2149069e48dcf5e564595e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:24.850790 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bcd7dec9c2149069e48dcf5e564595e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:24.851488 2053 leader_election.cc:343] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:24.851754 2053 leader_election.cc:343] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.851917 2053 leader_election.cc:304] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:24.852157 2381 ts_tablet_manager.cc:1434] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251026 02:56:24.852376 2506 raft_consensus.cc:2749] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:24.854357 2376 ts_tablet_manager.cc:1929] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.854480 2376 log.cc:1199] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/b435520cadb344edaa29975b4e19718a
I20251026 02:56:24.854691 2381 tablet_bootstrap.cc:492] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:24.854825 2375 tablet_bootstrap.cc:492] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:24.854877 2376 ts_tablet_manager.cc:1950] T b435520cadb344edaa29975b4e19718a P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:24.854981 2375 ts_tablet_manager.cc:1403] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20251026 02:56:24.859149 2405 consensus_queue.cc:1048] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:24.860782 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet d8ae08997af74b66b7d6375212eee4c2 (table default.Table_c095574955cf44d7aecfbca77de4ad1f [id=789c86929bf94d2ca3fb5289f598e292]). Expected version 1 got 0
W20251026 02:56:24.861663 2195 consensus_peers.cc:597] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bc92f90a68b4e749bfd4fa49076bfe2. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:24.863649 2087 tablet_service.cc:1505] Processing CreateTablet for tablet efd751ee4b1d433198dcd00c081f648d (DEFAULT_TABLE table=default.Table_133a36cbb31242b680078785eb21866e [id=20d9432d9f6443099d7fc0acb952a839]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.864112 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efd751ee4b1d433198dcd00c081f648d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.867581 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 3327d26a2c2c4d0b86d98151f5940bd7 (DEFAULT_TABLE table=default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 [id=7de20c0869584b56b936144d62202444]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.868081 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3327d26a2c2c4d0b86d98151f5940bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.868352 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 7062c5ff91e14369ab69a2d199585c55 (DEFAULT_TABLE table=default.Table_d48e174e391f42d6aa0b05c322b5b828 [id=91a568329da84558b31575e5127f873f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.868517 2092 tablet_service.cc:1505] Processing CreateTablet for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (DEFAULT_TABLE table=default.Table_87f51d8912864f0aa7ef8b490392611e [id=2e75e06786d940d092bdc81772be0dd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.868794 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7062c5ff91e14369ab69a2d199585c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.869072 2084 tablet_service.cc:1505] Processing CreateTablet for tablet cfc3aed64ad0439cb22fd6815c973d87 (DEFAULT_TABLE table=default.Table_34881d9847d5463297830bf6cc4ed1e6 [id=a484f36091f24fd2b8bc2af4f53eb450]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.869437 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfc3aed64ad0439cb22fd6815c973d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.873654 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1ab21ddf8e45bc9063b63db9c1b97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.875514 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet 8dbc12d49b094dbeadad2e76916ba9af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.876523 2506 raft_consensus.cc:493] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:24.876618 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet ea910e98d17340efb816a8aa90f0ba7e with delete_type TABLET_DATA_DELETED (replaced by tablet 1370b9ad917344c093e5da3d0e9b1cf0) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.876715 2506 raft_consensus.cc:515] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.877436 2506 leader_election.cc:290] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:24.878400 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d (DEFAULT_TABLE table=default.Table_7b08384413864da0a28224dd7201c263 [id=a9255e57e54144ba876815cbb6155360]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.878451 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3791cb18fe1f4937badb4859d00647f3" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:24.878860 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.878895 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet 115ccccf16b74efc9f24b607cf0c74d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:24.879339 2053 leader_election.cc:343] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.879969 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 115ccccf16b74efc9f24b607cf0c74d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.881096 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet fd1484434dd64f4bb2764268b3d3b218 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.883026 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3791cb18fe1f4937badb4859d00647f3" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:24.883266 2262 raft_consensus.cc:2468] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:24.883692 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 8dbc12d49b094dbeadad2e76916ba9af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.884480 2376 tablet_replica.cc:333] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:24.884997 2376 raft_consensus.cc:2243] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.885190 2372 tablet_replica.cc:333] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:24.885459 2372 raft_consensus.cc:2243] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:24.885697 2372 raft_consensus.cc:2272] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.885713 2376 raft_consensus.cc:2272] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.887732 2372 ts_tablet_manager.cc:1916] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.855795 2375 raft_consensus.cc:359] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.889588 2375 raft_consensus.cc:385] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.889683 2375 raft_consensus.cc:740] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.890307 2381 tablet_bootstrap.cc:654] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.886605 2053 leader_election.cc:304] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:24.894822 2376 ts_tablet_manager.cc:1916] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.895787 2378 tablet_bootstrap.cc:492] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:24.896101 2378 ts_tablet_manager.cc:1403] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.142s user 0.006s sys 0.014s
I20251026 02:56:24.898512 2378 raft_consensus.cc:359] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.898732 2378 raft_consensus.cc:385] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.898799 2377 ts_tablet_manager.cc:1929] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.898841 2378 raft_consensus.cc:740] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.899117 2378 consensus_queue.cc:260] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.900635 2378 ts_tablet_manager.cc:1434] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:24.902150 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 81530bbf4a2d4e908add683475e594fb (table default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]). Expected version 1 got 0
I20251026 02:56:24.910003 1656 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]). Expected version 2 got 0
I20251026 02:56:24.910544 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet ad55040f0a2b4cfa894d1bd058d4d69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.911727 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 17d2cc6b16d34331bb80c576c0cd2ce0) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.912627 2375 consensus_queue.cc:260] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.913089 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (DEFAULT_TABLE table=default.Table_7be811f036924663898ad10ce19955b1 [id=44c37945ff9f48e8b2c07648394ac177]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.913544 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.914354 2378 tablet_bootstrap.cc:492] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:24.915359 2096 tablet_service.cc:1505] Processing CreateTablet for tablet ce922040298d4952af13652eeda7a234 (DEFAULT_TABLE table=default.Table_0ea3c4e29f264b34b9f6030b92262af6 [id=d33bf698f19d407da868aa876f64de5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.915596 2506 raft_consensus.cc:2804] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:24.915719 2506 raft_consensus.cc:493] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:24.915814 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce922040298d4952af13652eeda7a234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.917357 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e56b259a314422aba35eb129021bbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.917475 2506 raft_consensus.cc:3060] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:24.918483 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet f8533f1c191042ee9f2c0b99972fe931 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.919018 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet cfed3523205548508d1101586ee3e2a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 44af74033ccf4d048475f12d164938f5) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:24.920692 1656 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 81530bbf4a2d4e908add683475e594fb (table default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]). Expected version 1 got 0
I20251026 02:56:24.921031 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 2487bf7d40f34922aa2fd165488342f6 (table default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]) successfully deleted
I20251026 02:56:24.921310 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 0e89c69e3b58458aad10beaeb5128d60 (table default.Table_00dc4c49a19246889353df2157eebf15 [id=347d712aca4e477c89805846800fb824]) successfully deleted
I20251026 02:56:24.921571 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet b435520cadb344edaa29975b4e19718a (table default.Table_66dd770512e847f4a14ee04e5afa3a6c [id=72fa811ef7224d42a83e602c3707f028]) successfully deleted
W20251026 02:56:24.921856 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0fe0e06a86e241bc91a1db4219fe25bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe0e06a86e241bc91a1db4219fe25bc already in progress: creating tablet
I20251026 02:56:24.922680 1953 tablet_service.cc:1505] Processing CreateTablet for tablet efd751ee4b1d433198dcd00c081f648d (DEFAULT_TABLE table=default.Table_133a36cbb31242b680078785eb21866e [id=20d9432d9f6443099d7fc0acb952a839]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.923099 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efd751ee4b1d433198dcd00c081f648d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.924680 2378 tablet_bootstrap.cc:654] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.924779 2376 ts_tablet_manager.cc:1929] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.924986 2376 log.cc:1199] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/829a6044a2da44ff90acdebca6d5b18d
I20251026 02:56:24.925508 2376 ts_tablet_manager.cc:1950] T 829a6044a2da44ff90acdebca6d5b18d P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:24.928143 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]). Expected version 2 got 0
I20251026 02:56:24.931829 2378 tablet_bootstrap.cc:492] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:24.932024 2378 ts_tablet_manager.cc:1403] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20251026 02:56:24.932883 2378 raft_consensus.cc:359] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.933085 2378 raft_consensus.cc:385] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.933183 2378 raft_consensus.cc:740] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.933427 2378 consensus_queue.cc:260] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.933427 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 47037db3602246a2ab6224557286162f (DEFAULT_TABLE table=default.Table_09ca38422af543f89eab26b4117536dc [id=12a8deac66914c6da970cd7f975b5b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.934118 2100 tablet_service.cc:1505] Processing CreateTablet for tablet a378e913112c443db869016e3b98fd0e (DEFAULT_TABLE table=default.Table_f2809b812fe14b2d9e44b22bfa68c1f7 [id=f83014c25cfa449d976ead67854d2ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.934517 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a378e913112c443db869016e3b98fd0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.934604 2378 ts_tablet_manager.cc:1434] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:24.934573 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 7062c5ff91e14369ab69a2d199585c55 (DEFAULT_TABLE table=default.Table_d48e174e391f42d6aa0b05c322b5b828 [id=91a568329da84558b31575e5127f873f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.935923 1957 tablet_service.cc:1505] Processing CreateTablet for tablet cfc3aed64ad0439cb22fd6815c973d87 (DEFAULT_TABLE table=default.Table_34881d9847d5463297830bf6cc4ed1e6 [id=a484f36091f24fd2b8bc2af4f53eb450]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.936389 2377 log.cc:1199] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7036a159e88b4171942dd238f8af3ed3
I20251026 02:56:24.936405 2372 ts_tablet_manager.cc:1929] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:24.909931 2405 consensus_queue.cc:1048] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:24.936585 2372 log.cc:1199] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/ea910e98d17340efb816a8aa90f0ba7e
I20251026 02:56:24.912341 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bc92f90a68b4e749bfd4fa49076bfe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 17d2cc6b16d34331bb80c576c0cd2ce0) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.916987 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (DEFAULT_TABLE table=default.Table_7be811f036924663898ad10ce19955b1 [id=44c37945ff9f48e8b2c07648394ac177]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.937464 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.937686 1951 tablet_service.cc:1505] Processing CreateTablet for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (DEFAULT_TABLE table=default.Table_87f51d8912864f0aa7ef8b490392611e [id=2e75e06786d940d092bdc81772be0dd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.917994 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e56b259a314422aba35eb129021bbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.940490 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1ab21ddf8e45bc9063b63db9c1b97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.919524 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet cfed3523205548508d1101586ee3e2a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 44af74033ccf4d048475f12d164938f5) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.919804 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet f8533f1c191042ee9f2c0b99972fe931 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:24.930774 2628 tablet.cc:1722] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:24.933097 2237 tablet_service.cc:1505] Processing CreateTablet for tablet efd751ee4b1d433198dcd00c081f648d (DEFAULT_TABLE table=default.Table_133a36cbb31242b680078785eb21866e [id=20d9432d9f6443099d7fc0acb952a839]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.944502 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efd751ee4b1d433198dcd00c081f648d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.945690 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47037db3602246a2ab6224557286162f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.946017 2372 ts_tablet_manager.cc:1950] T ea910e98d17340efb816a8aa90f0ba7e P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:24.935250 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 7062c5ff91e14369ab69a2d199585c55 (DEFAULT_TABLE table=default.Table_d48e174e391f42d6aa0b05c322b5b828 [id=91a568329da84558b31575e5127f873f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.948514 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 17d2cc6b16d34331bb80c576c0cd2ce0 (DEFAULT_TABLE table=default.Table_9f3815b1bf464a83a574e3075d065956 [id=095ad208529348d4aea2d4e2840e07c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.948904 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7062c5ff91e14369ab69a2d199585c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.949820 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d2cc6b16d34331bb80c576c0cd2ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.950367 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet ea910e98d17340efb816a8aa90f0ba7e (table default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]) successfully deleted
I20251026 02:56:24.951186 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfc3aed64ad0439cb22fd6815c973d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.951766 2506 raft_consensus.cc:515] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.952462 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7062c5ff91e14369ab69a2d199585c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.953177 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3791cb18fe1f4937badb4859d00647f3" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:24.953873 2378 tablet_bootstrap.cc:492] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:24.954356 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3791cb18fe1f4937badb4859d00647f3" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:24.954561 2262 raft_consensus.cc:3060] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:24.956779 2230 tablet_service.cc:1505] Processing CreateTablet for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (DEFAULT_TABLE table=default.Table_87f51d8912864f0aa7ef8b490392611e [id=2e75e06786d940d092bdc81772be0dd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.957194 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1ab21ddf8e45bc9063b63db9c1b97f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:24.958350 2053 leader_election.cc:343] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.959510 2262 raft_consensus.cc:2468] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:24.960495 2506 leader_election.cc:290] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:24.961928 2506 raft_consensus.cc:493] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:24.962127 2506 raft_consensus.cc:515] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.963153 2506 leader_election.cc:290] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:24.969060 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 3327d26a2c2c4d0b86d98151f5940bd7 (DEFAULT_TABLE table=default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 [id=7de20c0869584b56b936144d62202444]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.960155 2053 leader_election.cc:304] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:24.970976 2646 raft_consensus.cc:2804] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:24.971401 2646 raft_consensus.cc:697] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:24.972095 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3327d26a2c2c4d0b86d98151f5940bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.972185 2646 consensus_queue.cc:237] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.975483 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 7439c051ca634fbe8b511d47ccede2ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:24.975443 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7439c051ca634fbe8b511d47ccede2ff" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:24.976379 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7439c051ca634fbe8b511d47ccede2ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7439c051ca634fbe8b511d47ccede2ff already in progress: creating tablet
W20251026 02:56:24.976996 2053 leader_election.cc:343] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.979549 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d (DEFAULT_TABLE table=default.Table_7b08384413864da0a28224dd7201c263 [id=a9255e57e54144ba876815cbb6155360]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.979960 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.909376 2381 tablet_bootstrap.cc:492] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:24.936825 2235 tablet_service.cc:1505] Processing CreateTablet for tablet cfc3aed64ad0439cb22fd6815c973d87 (DEFAULT_TABLE table=default.Table_34881d9847d5463297830bf6cc4ed1e6 [id=a484f36091f24fd2b8bc2af4f53eb450]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.980625 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfc3aed64ad0439cb22fd6815c973d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.987524 2381 ts_tablet_manager.cc:1403] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.135s user 0.003s sys 0.001s
I20251026 02:56:24.988622 2381 raft_consensus.cc:359] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.988950 2381 raft_consensus.cc:385] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:24.989089 2381 raft_consensus.cc:740] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:24.989457 2381 consensus_queue.cc:260] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:24.990474 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 17d2cc6b16d34331bb80c576c0cd2ce0 (DEFAULT_TABLE table=default.Table_9f3815b1bf464a83a574e3075d065956 [id=095ad208529348d4aea2d4e2840e07c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.990875 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d2cc6b16d34331bb80c576c0cd2ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.993695 2381 ts_tablet_manager.cc:1434] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251026 02:56:24.993774 2375 ts_tablet_manager.cc:1434] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.139s user 0.007s sys 0.001s
I20251026 02:56:24.995868 2381 tablet_bootstrap.cc:492] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:24.998040 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 47037db3602246a2ab6224557286162f (DEFAULT_TABLE table=default.Table_09ca38422af543f89eab26b4117536dc [id=12a8deac66914c6da970cd7f975b5b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.998533 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47037db3602246a2ab6224557286162f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.999184 2375 tablet_bootstrap.cc:492] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.001765 2375 tablet_bootstrap.cc:654] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.003499 2236 tablet_service.cc:1505] Processing CreateTablet for tablet ce922040298d4952af13652eeda7a234 (DEFAULT_TABLE table=default.Table_0ea3c4e29f264b34b9f6030b92262af6 [id=d33bf698f19d407da868aa876f64de5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.003993 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce922040298d4952af13652eeda7a234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:24.955670 2378 tablet_bootstrap.cc:654] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.008397 2377 ts_tablet_manager.cc:1950] T 7036a159e88b4171942dd238f8af3ed3 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:24.966002 2648 tablet.cc:1722] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:25.009727 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 1c1596dd4394419a8bb6ea6a84be9fea (table default.Table_4a7c7fde2faf424590cdf3e2ddf0c1a5 [id=d9c0bfe3ef1145189ccef147212f787b]) successfully deleted
I20251026 02:56:25.010319 2377 tablet_replica.cc:333] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:25.010581 2377 raft_consensus.cc:2243] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.010668 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7036a159e88b4171942dd238f8af3ed3 (table default.Table_7e5c0b5970974e12aba66398ec59430e [id=59edc20bb9be438c9ef00aafcd1c6376]) successfully deleted
W20251026 02:56:25.010890 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0fe0e06a86e241bc91a1db4219fe25bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe0e06a86e241bc91a1db4219fe25bc already in progress: creating tablet
I20251026 02:56:25.010898 2377 raft_consensus.cc:2272] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.012127 2377 ts_tablet_manager.cc:1916] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:24.975322 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7439c051ca634fbe8b511d47ccede2ff" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:25.015329 2053 leader_election.cc:343] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:24.978752 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d (DEFAULT_TABLE table=default.Table_7b08384413864da0a28224dd7201c263 [id=a9255e57e54144ba876815cbb6155360]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.015497 2053 leader_election.cc:304] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:25.015854 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.015966 2506 raft_consensus.cc:2749] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:25.000128 1962 tablet_service.cc:1505] Processing CreateTablet for tablet ce922040298d4952af13652eeda7a234 (DEFAULT_TABLE table=default.Table_0ea3c4e29f264b34b9f6030b92262af6 [id=d33bf698f19d407da868aa876f64de5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.016780 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce922040298d4952af13652eeda7a234. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:25.020408 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 115ccccf16b74efc9f24b607cf0c74d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 115ccccf16b74efc9f24b607cf0c74d1 already in progress: creating tablet
I20251026 02:56:25.021337 1961 tablet_service.cc:1505] Processing CreateTablet for tablet a378e913112c443db869016e3b98fd0e (DEFAULT_TABLE table=default.Table_f2809b812fe14b2d9e44b22bfa68c1f7 [id=f83014c25cfa449d976ead67854d2ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.021736 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a378e913112c443db869016e3b98fd0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.024849 2237 tablet_service.cc:1505] Processing CreateTablet for tablet a378e913112c443db869016e3b98fd0e (DEFAULT_TABLE table=default.Table_f2809b812fe14b2d9e44b22bfa68c1f7 [id=f83014c25cfa449d976ead67854d2ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.025465 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a378e913112c443db869016e3b98fd0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.031643 2485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 02:56:24.998528 2381 tablet_bootstrap.cc:654] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:24.998215 2376 tablet_replica.cc:333] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:25.007901 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 17d2cc6b16d34331bb80c576c0cd2ce0 (DEFAULT_TABLE table=default.Table_9f3815b1bf464a83a574e3075d065956 [id=095ad208529348d4aea2d4e2840e07c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.032016 2376 raft_consensus.cc:2243] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.032241 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d2cc6b16d34331bb80c576c0cd2ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.032269 2376 raft_consensus.cc:2272] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:24.997181 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 47037db3602246a2ab6224557286162f (DEFAULT_TABLE table=default.Table_09ca38422af543f89eab26b4117536dc [id=12a8deac66914c6da970cd7f975b5b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:24.968287 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 3327d26a2c2c4d0b86d98151f5940bd7 (DEFAULT_TABLE table=default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 [id=7de20c0869584b56b936144d62202444]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.034184 2376 ts_tablet_manager.cc:1916] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.034348 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3327d26a2c2c4d0b86d98151f5940bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.036759 2485 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 02:56:25.036948 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ee0a408253e4f2b942c510ae64c703e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:21 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.037236 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47037db3602246a2ab6224557286162f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.038396 2485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 02:56:25.040117 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 829a6044a2da44ff90acdebca6d5b18d (table default.Table_6546379855b749b889279973db1820cd [id=b8be7adba1cd48ae979de506d892cc3d]) successfully deleted
W20251026 02:56:25.040810 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 8dbc12d49b094dbeadad2e76916ba9af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dbc12d49b094dbeadad2e76916ba9af already in progress: creating tablet
W20251026 02:56:25.041057 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 115ccccf16b74efc9f24b607cf0c74d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 115ccccf16b74efc9f24b607cf0c74d1 already in progress: creating tablet
I20251026 02:56:25.042296 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_17afc2f758be4959a0fceca2a296e3e1" } new_table_name: "default.Table_4fd5f39751624015ad57dd7fc77ce18a" modify_external_catalogs: true
I20251026 02:56:25.046900 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9f25820b797a4156aa4bfd227c3a0087" } new_table_name: "default.Table_481575c8f05747c2954c9c79a0ebde9c" modify_external_catalogs: true
I20251026 02:56:25.048864 2375 tablet_bootstrap.cc:492] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.049038 2375 ts_tablet_manager.cc:1403] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.011s
I20251026 02:56:25.048928 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_914e56e1fb53491a85ccbc64fb2b0769"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.049626 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_914e56e1fb53491a85ccbc64fb2b0769 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.049834 2375 raft_consensus.cc:359] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.049988 2375 raft_consensus.cc:385] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.050055 2375 raft_consensus.cc:740] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.050276 2375 consensus_queue.cc:260] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.051527 2375 ts_tablet_manager.cc:1434] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:25.052667 2375 tablet_bootstrap.cc:492] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.053189 2377 ts_tablet_manager.cc:1929] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.053339 2377 log.cc:1199] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/fd1484434dd64f4bb2764268b3d3b218
I20251026 02:56:25.054783 2377 ts_tablet_manager.cc:1950] T fd1484434dd64f4bb2764268b3d3b218 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:25.054872 2375 tablet_bootstrap.cc:654] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.055850 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet fd1484434dd64f4bb2764268b3d3b218 (table default.Table_f43d7cfe30cd40c08a88e4a62da5cbf4 [id=d342cbe7a7cb44928ec73415b30a779e]) successfully deleted
W20251026 02:56:25.057174 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8667ffa25f2b478a987826d486400bbe consensus state has no leader
I20251026 02:56:25.057520 2377 tablet_replica.cc:333] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:25.058275 2377 raft_consensus.cc:2243] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:25.058681 2377 raft_consensus.cc:2272] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.060186 2377 ts_tablet_manager.cc:1916] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.060196 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e4fbc44faf984982986abfb227d6663d" } modify_external_catalogs: true
I20251026 02:56:25.061764 2375 tablet_bootstrap.cc:492] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.061960 2375 ts_tablet_manager.cc:1403] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20251026 02:56:25.062824 2375 raft_consensus.cc:359] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.063036 2375 raft_consensus.cc:385] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.063112 2375 raft_consensus.cc:740] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.063421 2375 consensus_queue.cc:260] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:25.063486 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98fe44aab33b40eba819386c3bffca46 consensus state has no leader
I20251026 02:56:25.065512 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e4fbc44faf984982986abfb227d6663d" } modify_external_catalogs: true
I20251026 02:56:25.066944 2375 ts_tablet_manager.cc:1434] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20251026 02:56:25.068471 2375 tablet_bootstrap.cc:492] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.070546 2375 tablet_bootstrap.cc:654] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:25.071077 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 8dbc12d49b094dbeadad2e76916ba9af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dbc12d49b094dbeadad2e76916ba9af already in progress: creating tablet
I20251026 02:56:25.075764 2377 ts_tablet_manager.cc:1929] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:25.076022 2377 log.cc:1199] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7bc92f90a68b4e749bfd4fa49076bfe2
I20251026 02:56:25.076835 2377 ts_tablet_manager.cc:1950] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:25.077116 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d29c88d224ee40599643abab25ad8acd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.077781 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d29c88d224ee40599643abab25ad8acd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.081233 2375 tablet_bootstrap.cc:492] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.081411 2375 ts_tablet_manager.cc:1403] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20251026 02:56:25.082278 2375 raft_consensus.cc:359] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.082479 2375 raft_consensus.cc:385] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.082579 2375 raft_consensus.cc:740] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.082701 2393 raft_consensus.cc:493] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.082863 2375 consensus_queue.cc:260] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.084143 2375 ts_tablet_manager.cc:1434] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251026 02:56:25.084663 2375 tablet_bootstrap.cc:492] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.084580 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d4304259ac684b8097cdbf5511ab67d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.085196 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4304259ac684b8097cdbf5511ab67d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.085568 2393 raft_consensus.cc:515] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.086297 2393 leader_election.cc:290] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:25.087868 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_249435859b1e471585bf137537ac2446"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:25.088184 2375 tablet_bootstrap.cc:654] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:25.088537 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_249435859b1e471585bf137537ac2446 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:25.089525 1915 leader_election.cc:343] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 2ee0a408253e4f2b942c510ae64c703e
I20251026 02:56:25.091162 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_22ae79375e8e49d4b6d146d087ffa64b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.091670 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22ae79375e8e49d4b6d146d087ffa64b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:25.092527 1916 leader_election.cc:343] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 2ee0a408253e4f2b942c510ae64c703e
I20251026 02:56:25.092664 1916 leader_election.cc:304] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:25.095058 2376 ts_tablet_manager.cc:1929] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.095194 2376 log.cc:1199] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/ad55040f0a2b4cfa894d1bd058d4d69f
I20251026 02:56:25.095566 2376 ts_tablet_manager.cc:1950] T ad55040f0a2b4cfa894d1bd058d4d69f P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:25.096386 2393 raft_consensus.cc:2749] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:25.097815 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f2809b812fe14b2d9e44b22bfa68c1f7" } modify_external_catalogs: true
I20251026 02:56:25.098013 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f2809b812fe14b2d9e44b22bfa68c1f7" } modify_external_catalogs: true
I20251026 02:56:25.098632 2375 tablet_bootstrap.cc:492] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.098834 2375 ts_tablet_manager.cc:1403] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20251026 02:56:25.099699 2375 raft_consensus.cc:359] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.099928 2375 raft_consensus.cc:385] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.100021 2375 raft_consensus.cc:740] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.100345 2375 consensus_queue.cc:260] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.101104 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e2d286228a8244d2bded2ee992c5a064"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:25.101228 2375 ts_tablet_manager.cc:1434] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:25.101572 2375 tablet_bootstrap.cc:492] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:25.101673 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2d286228a8244d2bded2ee992c5a064 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.103435 2375 tablet_bootstrap.cc:654] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.104863 2376 tablet_replica.cc:333] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:25.104899 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3c56d6d5af8c44139ac28b8cc9945a9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:25.105074 2376 raft_consensus.cc:2243] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.105258 2376 raft_consensus.cc:2272] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:25.105470 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c56d6d5af8c44139ac28b8cc9945a9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.107128 2376 ts_tablet_manager.cc:1916] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.108880 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c095574955cf44d7aecfbca77de4ad1f" } modify_external_catalogs: true
I20251026 02:56:25.109081 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c095574955cf44d7aecfbca77de4ad1f" } modify_external_catalogs: true
I20251026 02:56:25.109879 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1a48da091e984e09aaffb9caed79f2c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.110390 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a48da091e984e09aaffb9caed79f2c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.110476 2646 raft_consensus.cc:493] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.110643 2646 raft_consensus.cc:515] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.111265 2646 leader_election.cc:290] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:25.112063 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20dec74440cf4bb9949c462ec51e9889" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:25.112792 2405 raft_consensus.cc:493] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.112954 2405 raft_consensus.cc:515] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.113013 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20dec74440cf4bb9949c462ec51e9889" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:25.113615 2405 leader_election.cc:290] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:25.113628 2053 leader_election.cc:343] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:25.113837 2053 leader_election.cc:343] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.113937 2053 leader_election.cc:304] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:25.114462 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98fe44aab33b40eba819386c3bffca46" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:25.114715 2117 raft_consensus.cc:2468] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:25.115083 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98fe44aab33b40eba819386c3bffca46" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:25.115185 2195 leader_election.cc:304] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:25.115533 2646 raft_consensus.cc:2749] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:25.115635 2194 leader_election.cc:343] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.115942 2405 raft_consensus.cc:2804] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:25.116048 2405 raft_consensus.cc:493] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:25.116132 2405 raft_consensus.cc:3060] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.118959 2405 raft_consensus.cc:515] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.126775 2378 tablet_bootstrap.cc:492] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:25.127007 2378 ts_tablet_manager.cc:1403] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.174s user 0.004s sys 0.001s
I20251026 02:56:25.127255 2405 leader_election.cc:290] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:25.114799 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.128245 2378 raft_consensus.cc:359] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.128500 2378 raft_consensus.cc:385] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.128620 2378 raft_consensus.cc:740] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.128912 2378 consensus_queue.cc:260] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.129199 2376 ts_tablet_manager.cc:1929] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:25.130431 2377 tablet_replica.cc:333] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:25.130479 2376 log.cc:1199] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/7bc92f90a68b4e749bfd4fa49076bfe2
I20251026 02:56:25.130606 2375 tablet_bootstrap.cc:492] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.130667 2377 raft_consensus.cc:2243] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.130764 2375 ts_tablet_manager.cc:1403] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.001s
I20251026 02:56:25.130841 2376 ts_tablet_manager.cc:1950] T 7bc92f90a68b4e749bfd4fa49076bfe2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:25.130852 2377 raft_consensus.cc:2272] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.131331 2378 ts_tablet_manager.cc:1434] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251026 02:56:25.131647 2377 ts_tablet_manager.cc:1916] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.131742 2376 tablet_replica.cc:333] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:25.131659 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98fe44aab33b40eba819386c3bffca46" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:25.131949 2117 raft_consensus.cc:3060] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.131659 2375 raft_consensus.cc:359] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.133324 2375 raft_consensus.cc:385] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.133425 2375 raft_consensus.cc:740] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.133680 2375 consensus_queue.cc:260] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.134253 2375 ts_tablet_manager.cc:1434] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 02:56:25.135946 2117 raft_consensus.cc:2468] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:25.136194 2375 tablet_bootstrap.cc:492] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.136399 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98fe44aab33b40eba819386c3bffca46" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:25.136461 2195 leader_election.cc:304] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:25.138756 2377 ts_tablet_manager.cc:1929] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251026 02:56:25.138852 2194 leader_election.cc:343] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.138993 2405 raft_consensus.cc:2804] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:25.139447 2405 raft_consensus.cc:697] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:25.139493 2375 tablet_bootstrap.cc:654] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.140257 2405 consensus_queue.cc:237] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.141494 2377 log.cc:1199] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3f60bcc5e4cb4fd19d2a5713f69814cd
I20251026 02:56:25.141901 2377 ts_tablet_manager.cc:1950] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:25.142942 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6fd0ab283f2c4b50a4e6cc4cb073c742"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.143754 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fd0ab283f2c4b50a4e6cc4cb073c742 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.146631 2376 raft_consensus.cc:2243] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.146931 2376 raft_consensus.cc:2272] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.147786 2376 ts_tablet_manager.cc:1916] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.148371 2381 tablet_bootstrap.cc:492] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.148643 2381 ts_tablet_manager.cc:1403] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.153s user 0.005s sys 0.000s
I20251026 02:56:25.149462 2381 raft_consensus.cc:359] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.149650 2381 raft_consensus.cc:385] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.149750 2381 raft_consensus.cc:740] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.149996 2381 consensus_queue.cc:260] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.153443 2381 ts_tablet_manager.cc:1434] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251026 02:56:25.154155 2381 tablet_bootstrap.cc:492] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.155913 2381 tablet_bootstrap.cc:654] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.159170 2378 tablet_bootstrap.cc:492] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:25.161674 2378 tablet_bootstrap.cc:654] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.163378 2376 ts_tablet_manager.cc:1929] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.163730 2376 log.cc:1199] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/2ee0a408253e4f2b942c510ae64c703e
I20251026 02:56:25.163849 1659 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]). Expected version 2 got 0
I20251026 02:56:25.164561 2376 ts_tablet_manager.cc:1950] T 2ee0a408253e4f2b942c510ae64c703e P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:25.164695 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]). Expected version 2 got 0
I20251026 02:56:25.165138 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8a12471ce48e486fa7959732288e0ff4" } new_table_name: "default.Table_4c889f76507c4e3e975fd19c21a2a37a" modify_external_catalogs: true
I20251026 02:56:25.165000 1661 catalog_manager.cc:5649] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:25.165259 2375 tablet_bootstrap.cc:492] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.165446 2375 ts_tablet_manager.cc:1403] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20251026 02:56:25.166076 2381 tablet_bootstrap.cc:492] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.166252 2381 ts_tablet_manager.cc:1403] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20251026 02:56:25.166433 2375 raft_consensus.cc:359] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.166676 2375 raft_consensus.cc:385] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.166767 2375 raft_consensus.cc:740] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.167202 2381 raft_consensus.cc:359] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.167394 2381 raft_consensus.cc:385] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.167469 2381 raft_consensus.cc:740] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.167557 1979 raft_consensus.cc:3060] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.168459 2381 consensus_queue.cc:260] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.169214 2381 ts_tablet_manager.cc:1434] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251026 02:56:25.169286 2375 consensus_queue.cc:260] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.169580 2381 tablet_bootstrap.cc:492] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.169843 2375 ts_tablet_manager.cc:1434] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:25.171777 2381 tablet_bootstrap.cc:654] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.173698 2375 tablet_bootstrap.cc:492] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.175617 2375 tablet_bootstrap.cc:654] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.179745 1654 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]). Expected version 2 got 0
I20251026 02:56:25.179986 2381 tablet_bootstrap.cc:492] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.180140 2381 ts_tablet_manager.cc:1403] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20251026 02:56:25.181218 2381 raft_consensus.cc:359] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.181401 2381 raft_consensus.cc:385] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.181494 2381 raft_consensus.cc:740] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.181712 2381 consensus_queue.cc:260] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.182363 2381 ts_tablet_manager.cc:1434] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:25.182652 2381 tablet_bootstrap.cc:492] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.184723 2381 tablet_bootstrap.cc:654] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:25.193648 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 049a915b35224cdaa38df77c734a0f01 consensus state has no leader
I20251026 02:56:25.194084 2378 tablet_bootstrap.cc:492] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:25.194388 2378 ts_tablet_manager.cc:1403] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20251026 02:56:25.195322 2378 raft_consensus.cc:359] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.195637 2378 raft_consensus.cc:385] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.195915 2378 raft_consensus.cc:740] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.196327 2378 consensus_queue.cc:260] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.196903 2375 tablet_bootstrap.cc:492] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.197059 2378 ts_tablet_manager.cc:1434] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:25.197085 2375 ts_tablet_manager.cc:1403] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.004s
I20251026 02:56:25.198390 2378 tablet_bootstrap.cc:492] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:25.199211 2643 raft_consensus.cc:493] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.199412 2643 raft_consensus.cc:515] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.199668 2375 raft_consensus.cc:359] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.199889 2375 raft_consensus.cc:385] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.199978 2375 raft_consensus.cc:740] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.200578 2643 leader_election.cc:290] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:25.201269 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8adf5c6970be46ed8434fd28404b2166" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:25.201457 2375 consensus_queue.cc:260] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.201692 2646 raft_consensus.cc:493] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:25.201864 2194 leader_election.cc:343] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.201867 2646 raft_consensus.cc:515] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.201421 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d197f56404544c8cb996d53edb9559a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:25.202077 2375 ts_tablet_manager.cc:1434] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20251026 02:56:25.202504 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d197f56404544c8cb996d53edb9559a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.202651 2646 leader_election.cc:290] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:25.203420 2375 tablet_bootstrap.cc:492] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.204543 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8667ffa25f2b478a987826d486400bbe" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:25.205447 2053 leader_election.cc:343] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.205960 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8adf5c6970be46ed8434fd28404b2166" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:25.206200 2117 raft_consensus.cc:2468] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:25.206986 2195 leader_election.cc:304] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:25.207507 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8667ffa25f2b478a987826d486400bbe" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:25.207959 2643 raft_consensus.cc:2804] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:25.208081 2643 raft_consensus.cc:493] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:25.208168 2643 raft_consensus.cc:3060] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.210767 2381 tablet_bootstrap.cc:492] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.210937 2381 ts_tablet_manager.cc:1403] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.003s
I20251026 02:56:25.211772 2381 raft_consensus.cc:359] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.211997 2381 raft_consensus.cc:385] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.212085 2381 raft_consensus.cc:740] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.212373 2381 consensus_queue.cc:260] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:25.212548 2053 leader_election.cc:343] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.212715 2053 leader_election.cc:304] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:25.213016 2381 ts_tablet_manager.cc:1434] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:25.213238 2375 tablet_bootstrap.cc:654] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.213287 2646 raft_consensus.cc:2749] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:25.213320 2381 tablet_bootstrap.cc:492] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.215415 2381 tablet_bootstrap.cc:654] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.215644 2643 raft_consensus.cc:515] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.216985 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8adf5c6970be46ed8434fd28404b2166" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:25.217248 2117 raft_consensus.cc:3060] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.219353 2375 tablet_bootstrap.cc:492] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.219544 2375 ts_tablet_manager.cc:1403] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20251026 02:56:25.220010 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 44af74033ccf4d048475f12d164938f5
I20251026 02:56:25.220477 2375 raft_consensus.cc:359] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.220702 2375 raft_consensus.cc:385] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.220785 2375 raft_consensus.cc:740] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.221329 2117 raft_consensus.cc:2468] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:25.222648 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8adf5c6970be46ed8434fd28404b2166" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:25.223480 2194 leader_election.cc:343] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.221479 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 44af74033ccf4d048475f12d164938f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.223299 2375 consensus_queue.cc:260] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.224040 2381 tablet_bootstrap.cc:492] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.224195 2381 ts_tablet_manager.cc:1403] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
W20251026 02:56:25.224411 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 44af74033ccf4d048475f12d164938f5 because the tablet was not found. No further retry: Not found: Tablet not found: 44af74033ccf4d048475f12d164938f5
I20251026 02:56:25.224644 2375 ts_tablet_manager.cc:1434] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251026 02:56:25.224994 2375 tablet_bootstrap.cc:492] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.224936 1654 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 23fd1284c9cb41bd98209ccbfe342bca (table default.Table_83bdf2cfef4042748553ebe8ae34996a [id=6734e5731cc94e71b43c76512f649d57]). Expected version 1 got 0
I20251026 02:56:25.225111 2381 raft_consensus.cc:359] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.225203 1654 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 8667ffa25f2b478a987826d486400bbe (table default.Table_4fd5f39751624015ad57dd7fc77ce18a [id=f7f1e852a46f4129b2709fd58a6be034]). Expected version 1 got 0
I20251026 02:56:25.225293 2381 raft_consensus.cc:385] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.225378 2381 raft_consensus.cc:740] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.225445 1654 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]). Expected version 1 got 0
I20251026 02:56:25.225610 2381 consensus_queue.cc:260] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.226270 2381 ts_tablet_manager.cc:1434] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:25.226598 2381 tablet_bootstrap.cc:492] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.228026 2195 leader_election.cc:304] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:25.228451 2631 raft_consensus.cc:2804] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:25.228693 2631 raft_consensus.cc:697] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:25.228739 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 7439c051ca634fbe8b511d47ccede2ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:25.229107 1654 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 23fd1284c9cb41bd98209ccbfe342bca (table default.Table_83bdf2cfef4042748553ebe8ae34996a [id=6734e5731cc94e71b43c76512f649d57], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 23fd1284c9cb41bd98209ccbfe342bca consensus state has no leader
I20251026 02:56:25.229007 2631 consensus_queue.cc:237] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.228505 2381 tablet_bootstrap.cc:654] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.229902 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 20dec74440cf4bb9949c462ec51e9889 with delete_type TABLET_DATA_DELETED (replaced by tablet c570e5f9a93c45bda32ec92c68794714) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.230516 2372 tablet_replica.cc:333] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:25.230576 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_914e56e1fb53491a85ccbc64fb2b0769" } new_table_name: "default.Table_a6e8714ddccb4139adb6c8320e06500b" modify_external_catalogs: true
I20251026 02:56:25.230721 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c647782049243d6bcd595f1a10d89c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 5caeb11322fe48348ebb523967152187) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.230746 2372 raft_consensus.cc:2243] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.230918 2372 raft_consensus.cc:2272] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.231729 2372 ts_tablet_manager.cc:1916] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.232324 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6546379855b749b889279973db1820cd" } modify_external_catalogs: true
I20251026 02:56:25.232395 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5edbe48bb8ce41de84eaeaa4b4137fdb" } new_table_name: "default.Table_424693529d0d4c829755adc2c9835c0e" modify_external_catalogs: true
I20251026 02:56:25.232520 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6546379855b749b889279973db1820cd" } modify_external_catalogs: true
I20251026 02:56:25.233775 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet b1a2522801364693be0bda6fa45a697e from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:25.234277 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 03da415f3a944a9a9679b3a45295ce35 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:25.234818 2375 tablet_bootstrap.cc:654] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.235481 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet 20dec74440cf4bb9949c462ec51e9889 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:25.235965 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet d8ae08997af74b66b7d6375212eee4c2 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:25.239210 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 44af74033ccf4d048475f12d164938f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.239368 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet ad55040f0a2b4cfa894d1bd058d4d69f (table default.Table_0da51595453549b3b7e077f7b7ff3ec7 [id=53cd517e68e64597ad51d85d16f0399d]) successfully deleted
I20251026 02:56:25.239668 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (table default.Table_9f3815b1bf464a83a574e3075d065956 [id=095ad208529348d4aea2d4e2840e07c7]) successfully deleted
W20251026 02:56:25.239905 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 1e56b259a314422aba35eb129021bbeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e56b259a314422aba35eb129021bbeb already in progress: creating tablet
I20251026 02:56:25.239993 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet f6d6f024d8b943d5acf69b64bc7c7214 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:25.240088 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet f8533f1c191042ee9f2c0b99972fe931 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8533f1c191042ee9f2c0b99972fe931 already in progress: creating tablet
I20251026 02:56:25.240926 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_689f802c9e4b457eb71fb78898fe14b0" } modify_external_catalogs: true
I20251026 02:56:25.241236 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_689f802c9e4b457eb71fb78898fe14b0" } modify_external_catalogs: true
I20251026 02:56:25.243307 1873 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 44af74033ccf4d048475f12d164938f5
I20251026 02:56:25.244235 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 44af74033ccf4d048475f12d164938f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.245106 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 44af74033ccf4d048475f12d164938f5 (DEFAULT_TABLE table=default.Table_e4fbc44faf984982986abfb227d6663d [id=b35a8189bbc04e94bb097c9e13a38ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.245275 2643 leader_election.cc:290] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:25.245529 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44af74033ccf4d048475f12d164938f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.246928 2381 tablet_bootstrap.cc:492] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.247112 2381 ts_tablet_manager.cc:1403] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20251026 02:56:25.247977 2381 raft_consensus.cc:359] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.248198 2381 raft_consensus.cc:385] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.248310 2381 raft_consensus.cc:740] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.248540 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 44af74033ccf4d048475f12d164938f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.248670 2381 consensus_queue.cc:260] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.249265 2381 ts_tablet_manager.cc:1434] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:25.249610 2381 tablet_bootstrap.cc:492] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.250008 2372 ts_tablet_manager.cc:1929] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.251088 2372 log.cc:1199] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7439c051ca634fbe8b511d47ccede2ff
I20251026 02:56:25.251492 2372 ts_tablet_manager.cc:1950] T 7439c051ca634fbe8b511d47ccede2ff P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:25.252039 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8688a0bf946841deb5d88c931538b129 has no consensus state
I20251026 02:56:25.253441 2378 tablet_bootstrap.cc:654] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.255321 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7439c051ca634fbe8b511d47ccede2ff (table default.Table_be7d5ddd5e4e4b15862de09dc573f7f9 [id=354a2ac9115e4f2385537f1c16992882]) successfully deleted
I20251026 02:56:25.255750 2372 tablet_replica.cc:333] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:25.256011 2372 raft_consensus.cc:2243] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.256935 2648 tablet.cc:1722] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251026 02:56:25.261467 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d8cdf79a97149618bcaf0fefee1dfeb consensus state has no leader
I20251026 02:56:25.264843 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 2ee0a408253e4f2b942c510ae64c703e (table default.Table_c550a959e98a4e22853deca526de3eae [id=47bcd084e7fa4dec9f23cef12d10aa46]) successfully deleted
I20251026 02:56:25.271394 2372 raft_consensus.cc:2272] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.272387 2372 ts_tablet_manager.cc:1916] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.273736 2381 tablet_bootstrap.cc:654] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:25.280539 1647 catalog_manager.cc:4952] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 44af74033ccf4d048475f12d164938f5 because the tablet was not found. No further retry: Not found: Tablet not found: 44af74033ccf4d048475f12d164938f5
I20251026 02:56:25.280980 2375 tablet_bootstrap.cc:492] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.281147 2375 ts_tablet_manager.cc:1403] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.007s
I20251026 02:56:25.281908 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0de06ea2b0544402a979d9ddc487860f
I20251026 02:56:25.282356 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 03da415f3a944a9a9679b3a45295ce35
I20251026 02:56:25.287972 1658 catalog_manager.cc:6719] ReplaceTablet: tablet 03da415f3a944a9a9679b3a45295ce35 deleted and replaced by tablet a6f9ae327aa4401baf66a737eaa93cf6
I20251026 02:56:25.282618 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a378e913112c443db869016e3b98fd0e
I20251026 02:56:25.283581 1660 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 81530bbf4a2d4e908add683475e594fb (table default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]). Expected version 1 got 0
I20251026 02:56:25.288996 2375 raft_consensus.cc:359] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.289072 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 0de06ea2b0544402a979d9ddc487860f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.289203 2375 raft_consensus.cc:385] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.289302 2375 raft_consensus.cc:740] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.289412 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 03da415f3a944a9a9679b3a45295ce35 with delete_type TABLET_DATA_DELETED (replaced by tablet a6f9ae327aa4401baf66a737eaa93cf6) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.289569 2375 consensus_queue.cc:260] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.290118 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7bc92f90a68b4e749bfd4fa49076bfe2 (table default.Table_9f3815b1bf464a83a574e3075d065956 [id=095ad208529348d4aea2d4e2840e07c7]) successfully deleted
W20251026 02:56:25.290412 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 1e56b259a314422aba35eb129021bbeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e56b259a314422aba35eb129021bbeb already in progress: creating tablet
W20251026 02:56:25.290784 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet f8533f1c191042ee9f2c0b99972fe931 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8533f1c191042ee9f2c0b99972fe931 already in progress: creating tablet
I20251026 02:56:25.290859 2375 ts_tablet_manager.cc:1434] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20251026 02:56:25.292815 2375 tablet_bootstrap.cc:492] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.294682 2375 tablet_bootstrap.cc:654] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.296180 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet a378e913112c443db869016e3b98fd0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.299338 2372 ts_tablet_manager.cc:1929] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.299896 2372 log.cc:1199] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/20dec74440cf4bb9949c462ec51e9889
I20251026 02:56:25.300575 2372 ts_tablet_manager.cc:1950] T 20dec74440cf4bb9949c462ec51e9889 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:25.291889 1654 catalog_manager.cc:5649] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:25.303332 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 98fe44aab33b40eba819386c3bffca46 (table default.Table_481575c8f05747c2954c9c79a0ebde9c [id=91d789aff2464f73b60e03b52126a396]). Expected version 1 got 0
I20251026 02:56:25.303426 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 20dec74440cf4bb9949c462ec51e9889 (table default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]) successfully deleted
I20251026 02:56:25.303351 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3f60bcc5e4cb4fd19d2a5713f69814cd (table default.Table_d029698b06db4a6f807451acb73a3680 [id=2ae19615aa104e2b8791b86755e1184f]) successfully deleted
I20251026 02:56:25.303853 2372 tablet_replica.cc:333] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:25.303906 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 44af74033ccf4d048475f12d164938f5 because the tablet was not found. No further retry: Not found: Tablet not found: 44af74033ccf4d048475f12d164938f5
I20251026 02:56:25.304104 2372 raft_consensus.cc:2243] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.304315 2372 raft_consensus.cc:2272] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.295367 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b1a2522801364693be0bda6fa45a697e
I20251026 02:56:25.304914 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 44af74033ccf4d048475f12d164938f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.304999 2372 ts_tablet_manager.cc:1916] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.305398 1661 catalog_manager.cc:6719] ReplaceTablet: tablet b1a2522801364693be0bda6fa45a697e deleted and replaced by tablet ff242ef9541a42d793c4484190956da6
I20251026 02:56:25.305866 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 44af74033ccf4d048475f12d164938f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.306210 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet b1a2522801364693be0bda6fa45a697e with delete_type TABLET_DATA_DELETED (replaced by tablet ff242ef9541a42d793c4484190956da6) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.306587 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 44af74033ccf4d048475f12d164938f5 (DEFAULT_TABLE table=default.Table_e4fbc44faf984982986abfb227d6663d [id=b35a8189bbc04e94bb097c9e13a38ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.307041 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44af74033ccf4d048475f12d164938f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.314364 2372 ts_tablet_manager.cc:1929] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.314599 2372 log.cc:1199] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/0c647782049243d6bcd595f1a10d89c6
I20251026 02:56:25.314872 2378 tablet_bootstrap.cc:492] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:25.315047 2378 ts_tablet_manager.cc:1403] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.117s user 0.008s sys 0.004s
I20251026 02:56:25.315073 2372 ts_tablet_manager.cc:1950] T 0c647782049243d6bcd595f1a10d89c6 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:25.315841 2378 raft_consensus.cc:359] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.316129 2378 raft_consensus.cc:385] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.316258 2378 raft_consensus.cc:740] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.316394 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 0c647782049243d6bcd595f1a10d89c6 (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
I20251026 02:56:25.316531 2378 consensus_queue.cc:260] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:25.317018 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 44af74033ccf4d048475f12d164938f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44af74033ccf4d048475f12d164938f5 already in progress: creating tablet
I20251026 02:56:25.317168 2378 ts_tablet_manager.cc:1434] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:25.317818 2378 tablet_bootstrap.cc:492] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:25.320497 2378 tablet_bootstrap.cc:654] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:25.323942 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0de06ea2b0544402a979d9ddc487860f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0de06ea2b0544402a979d9ddc487860f already in progress: creating tablet
I20251026 02:56:25.325558 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet d8ae08997af74b66b7d6375212eee4c2
W20251026 02:56:25.326805 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 03da415f3a944a9a9679b3a45295ce35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03da415f3a944a9a9679b3a45295ce35 already in progress: creating tablet
I20251026 02:56:25.327008 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ae08997af74b66b7d6375212eee4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:25.327046 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a378e913112c443db869016e3b98fd0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a378e913112c443db869016e3b98fd0e already in progress: creating tablet
W20251026 02:56:25.327239 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet b1a2522801364693be0bda6fa45a697e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1a2522801364693be0bda6fa45a697e already in progress: creating tablet
I20251026 02:56:25.327445 2372 tablet_replica.cc:333] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:25.327687 2372 raft_consensus.cc:2243] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.327723 2405 raft_consensus.cc:493] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.327987 2372 raft_consensus.cc:2272] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.327903 2405 raft_consensus.cc:515] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.328789 2405 leader_election.cc:290] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:25.329272 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bcd7dec9c2149069e48dcf5e564595e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:25.329371 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bcd7dec9c2149069e48dcf5e564595e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:25.329499 1979 raft_consensus.cc:2468] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:25.329584 2117 raft_consensus.cc:2468] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:25.329936 2194 leader_election.cc:304] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:25.328706 2372 ts_tablet_manager.cc:1916] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.330368 2405 raft_consensus.cc:2804] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:25.330469 2405 raft_consensus.cc:493] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:25.330583 2405 raft_consensus.cc:3060] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.331590 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 44af74033ccf4d048475f12d164938f5 (DEFAULT_TABLE table=default.Table_e4fbc44faf984982986abfb227d6663d [id=b35a8189bbc04e94bb097c9e13a38ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.332064 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44af74033ccf4d048475f12d164938f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.333263 2375 tablet_bootstrap.cc:492] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.333437 2375 ts_tablet_manager.cc:1403] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.011s
I20251026 02:56:25.334288 2375 raft_consensus.cc:359] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.334517 2375 raft_consensus.cc:385] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.334592 2375 raft_consensus.cc:740] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.334839 2375 consensus_queue.cc:260] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.335392 2375 ts_tablet_manager.cc:1434] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:25.335747 2375 tablet_bootstrap.cc:492] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.338506 2375 tablet_bootstrap.cc:654] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.340260 2405 raft_consensus.cc:515] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.341434 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bcd7dec9c2149069e48dcf5e564595e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:25.341667 1979 raft_consensus.cc:3060] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.342718 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 03da415f3a944a9a9679b3a45295ce35 with delete_type TABLET_DATA_DELETED (replaced by tablet a6f9ae327aa4401baf66a737eaa93cf6) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.344092 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 0de06ea2b0544402a979d9ddc487860f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.344991 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 03da415f3a944a9a9679b3a45295ce35 with delete_type TABLET_DATA_DELETED (replaced by tablet a6f9ae327aa4401baf66a737eaa93cf6) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.345244 1979 raft_consensus.cc:2468] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:25.345546 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 0de06ea2b0544402a979d9ddc487860f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.346246 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bcd7dec9c2149069e48dcf5e564595e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:25.346467 2120 raft_consensus.cc:3060] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.346918 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet a378e913112c443db869016e3b98fd0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.347190 2643 raft_consensus.cc:493] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.347384 2643 raft_consensus.cc:515] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.348171 2643 leader_election.cc:290] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:25.349254 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23fd1284c9cb41bd98209ccbfe342bca" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:25.349301 2120 raft_consensus.cc:2468] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:25.349519 2118 raft_consensus.cc:2468] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:25.349794 2195 leader_election.cc:304] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:25.350116 2195 leader_election.cc:304] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:25.350382 2440 raft_consensus.cc:2804] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:25.350544 2440 raft_consensus.cc:697] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:25.350828 2440 consensus_queue.cc:237] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.351281 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 86d8e1e21c5f440d8baf76c357d01ad2
I20251026 02:56:25.351820 2631 raft_consensus.cc:2804] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:25.351962 2631 raft_consensus.cc:493] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:25.352047 2631 raft_consensus.cc:3060] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.352104 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23fd1284c9cb41bd98209ccbfe342bca" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:25.352630 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 86d8e1e21c5f440d8baf76c357d01ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:25.352929 2194 leader_election.cc:343] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.354133 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 0020cb1f190c49978c5322d387a47f71 with delete_type TABLET_DATA_DELETED (replaced by tablet 86d8e1e21c5f440d8baf76c357d01ad2) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.354709 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 7176cbe8f64546b7871d047c0e0e5b2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.354908 2631 raft_consensus.cc:515] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.355489 2631 leader_election.cc:290] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:25.356832 2628 tablet.cc:1722] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:25.358707 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23fd1284c9cb41bd98209ccbfe342bca" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:25.358970 2117 raft_consensus.cc:3060] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.360201 2381 tablet_bootstrap.cc:492] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.360390 2381 ts_tablet_manager.cc:1403] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.111s user 0.009s sys 0.014s
I20251026 02:56:25.360513 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d29c88d224ee40599643abab25ad8acd" } new_table_name: "default.Table_1fba8934d98d4fdb8b2c9a0b5d01e810" modify_external_catalogs: true
I20251026 02:56:25.361310 2381 raft_consensus.cc:359] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.361507 2381 raft_consensus.cc:385] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.361588 2381 raft_consensus.cc:740] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.361778 2381 consensus_queue.cc:260] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.362093 2117 raft_consensus.cc:2468] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:25.362123 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d300a4e488f043afa43e544db231cc02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.362839 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d300a4e488f043afa43e544db231cc02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.363651 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet cfed3523205548508d1101586ee3e2a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 44af74033ccf4d048475f12d164938f5) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.363782 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 461412f3ad1e4186b1302cdf5a624ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.364368 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_42e1adcd9ad84e029d7706cc7fd847a1" } new_table_name: "default.Table_1361fbf89aa04321a7876c1e43e27089" modify_external_catalogs: true
I20251026 02:56:25.364877 2195 leader_election.cc:304] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:25.365057 2381 ts_tablet_manager.cc:1434] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251026 02:56:25.365410 2381 tablet_bootstrap.cc:492] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.365615 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet a378e913112c443db869016e3b98fd0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.365871 2643 raft_consensus.cc:2804] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:25.365965 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet fd1484434dd64f4bb2764268b3d3b218 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.366093 2643 raft_consensus.cc:697] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:25.366247 2376 tablet_replica.cc:333] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:25.366503 2376 raft_consensus.cc:2243] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.366489 2643 consensus_queue.cc:237] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.366706 2376 raft_consensus.cc:2272] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.366716 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet b1a2522801364693be0bda6fa45a697e with delete_type TABLET_DATA_DELETED (replaced by tablet ff242ef9541a42d793c4484190956da6) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.367836 2376 ts_tablet_manager.cc:1916] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.367987 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23fd1284c9cb41bd98209ccbfe342bca" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:25.372362 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet f6d6f024d8b943d5acf69b64bc7c7214
I20251026 02:56:25.372906 1659 catalog_manager.cc:6719] ReplaceTablet: tablet f6d6f024d8b943d5acf69b64bc7c7214 deleted and replaced by tablet 6a3491b303f94fe2a2c838297f73d464
I20251026 02:56:25.374737 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet f6d6f024d8b943d5acf69b64bc7c7214 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a3491b303f94fe2a2c838297f73d464) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.376117 2376 ts_tablet_manager.cc:1929] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.376607 2376 log.cc:1199] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/fd1484434dd64f4bb2764268b3d3b218
I20251026 02:56:25.377368 2376 ts_tablet_manager.cc:1950] T fd1484434dd64f4bb2764268b3d3b218 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:25.377856 2381 tablet_bootstrap.cc:654] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.380779 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ae08997af74b66b7d6375212eee4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.380525 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ae08997af74b66b7d6375212eee4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.381130 2377 tablet_replica.cc:333] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:25.381402 2377 raft_consensus.cc:2243] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.381639 2377 raft_consensus.cc:2272] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.381727 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet b1a2522801364693be0bda6fa45a697e with delete_type TABLET_DATA_DELETED (replaced by tablet ff242ef9541a42d793c4484190956da6) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.382336 2377 ts_tablet_manager.cc:1916] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.384739 2661 raft_consensus.cc:493] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.384912 2661 raft_consensus.cc:515] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.385908 2661 leader_election.cc:290] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:25.386479 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3791cb18fe1f4937badb4859d00647f3" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:25.387485 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 86d8e1e21c5f440d8baf76c357d01ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.387879 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3791cb18fe1f4937badb4859d00647f3" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:25.388141 2262 raft_consensus.cc:2393] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 0b6b0a6c4e144982b67980118cddba48 in this term.
I20251026 02:56:25.388760 2405 leader_election.cc:290] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:25.388744 1915 leader_election.cc:304] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:25.389178 2661 raft_consensus.cc:3060] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.382308 2381 tablet_bootstrap.cc:492] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.392597 2381 ts_tablet_manager.cc:1403] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20251026 02:56:25.393465 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 86d8e1e21c5f440d8baf76c357d01ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.393596 2381 raft_consensus.cc:359] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.393826 2381 raft_consensus.cc:385] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.393914 2381 raft_consensus.cc:740] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.394183 2381 consensus_queue.cc:260] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.394762 2381 ts_tablet_manager.cc:1434] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:25.394838 2646 raft_consensus.cc:493] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.394834 2661 raft_consensus.cc:2749] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:25.395152 2381 tablet_bootstrap.cc:492] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.395084 2646 raft_consensus.cc:515] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.395788 2646 leader_election.cc:290] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:25.395953 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 44af74033ccf4d048475f12d164938f5 because the tablet was not found. No further retry: Not found: Tablet not found: 44af74033ccf4d048475f12d164938f5
W20251026 02:56:25.396394 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 03da415f3a944a9a9679b3a45295ce35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03da415f3a944a9a9679b3a45295ce35 already in progress: creating tablet
W20251026 02:56:25.396648 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0de06ea2b0544402a979d9ddc487860f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0de06ea2b0544402a979d9ddc487860f already in progress: creating tablet
W20251026 02:56:25.396986 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet a378e913112c443db869016e3b98fd0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a378e913112c443db869016e3b98fd0e already in progress: creating tablet
W20251026 02:56:25.397302 1647 catalog_manager.cc:4952] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 44af74033ccf4d048475f12d164938f5 because the tablet was not found. No further retry: Not found: Tablet not found: 44af74033ccf4d048475f12d164938f5
W20251026 02:56:25.397616 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 03da415f3a944a9a9679b3a45295ce35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03da415f3a944a9a9679b3a45295ce35 already in progress: creating tablet
W20251026 02:56:25.397871 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0de06ea2b0544402a979d9ddc487860f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0de06ea2b0544402a979d9ddc487860f already in progress: creating tablet
W20251026 02:56:25.398126 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet a378e913112c443db869016e3b98fd0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a378e913112c443db869016e3b98fd0e already in progress: creating tablet
I20251026 02:56:25.398291 2372 ts_tablet_manager.cc:1929] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.398325 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet fd1484434dd64f4bb2764268b3d3b218 (table default.Table_f43d7cfe30cd40c08a88e4a62da5cbf4 [id=d342cbe7a7cb44928ec73415b30a779e]) successfully deleted
I20251026 02:56:25.398497 2372 log.cc:1199] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/d8ae08997af74b66b7d6375212eee4c2
W20251026 02:56:25.398555 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b1a2522801364693be0bda6fa45a697e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1a2522801364693be0bda6fa45a697e already in progress: creating tablet
W20251026 02:56:25.398751 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet d8ae08997af74b66b7d6375212eee4c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ae08997af74b66b7d6375212eee4c2 already in progress: creating tablet
I20251026 02:56:25.398999 2372 ts_tablet_manager.cc:1950] T d8ae08997af74b66b7d6375212eee4c2 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:25.399327 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet f6d6f024d8b943d5acf69b64bc7c7214 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a3491b303f94fe2a2c838297f73d464) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.400274 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ec035b1caac450d9ab55e74a061ae38" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:25.400914 2053 leader_election.cc:343] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.401824 2687 tablet.cc:1722] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251026 02:56:25.406276 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c3050e72d42417ab58a05f7e3aa56b3 has no consensus state
I20251026 02:56:25.407232 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet d8ae08997af74b66b7d6375212eee4c2 (table default.Table_c095574955cf44d7aecfbca77de4ad1f [id=789c86929bf94d2ca3fb5289f598e292]) successfully deleted
W20251026 02:56:25.407573 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 86d8e1e21c5f440d8baf76c357d01ad2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86d8e1e21c5f440d8baf76c357d01ad2 already in progress: creating tablet
I20251026 02:56:25.407582 2372 tablet_replica.cc:333] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:25.407912 2372 raft_consensus.cc:2243] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.408190 2372 raft_consensus.cc:2272] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.409053 2372 ts_tablet_manager.cc:1916] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.414849 2378 tablet_bootstrap.cc:492] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:25.415088 2378 ts_tablet_manager.cc:1403] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.098s user 0.001s sys 0.004s
I20251026 02:56:25.415647 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ec035b1caac450d9ab55e74a061ae38" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:25.416116 2378 raft_consensus.cc:359] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.416435 2378 raft_consensus.cc:385] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.416520 2378 raft_consensus.cc:740] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.416817 2378 consensus_queue.cc:260] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.417404 2378 ts_tablet_manager.cc:1434] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:25.417798 2378 tablet_bootstrap.cc:492] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:25.419687 2378 tablet_bootstrap.cc:654] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:25.423022 2053 leader_election.cc:343] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.423174 2053 leader_election.cc:304] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:25.423607 2646 raft_consensus.cc:2749] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:25.428447 2194 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 60332.4us
W20251026 02:56:25.429556 2194 leader_election.cc:343] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.430135 2372 ts_tablet_manager.cc:1929] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.430469 2372 log.cc:1199] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/0020cb1f190c49978c5322d387a47f71
I20251026 02:56:25.431002 2372 ts_tablet_manager.cc:1950] T 0020cb1f190c49978c5322d387a47f71 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:25.431548 1650 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 276bd3b97c97426daf802e1f57feb265 (table default.Table_6e87e5c48f4a4aa8be0b0ecc837e20dd [id=e91358e10bab4589a6f4e815a85a5f7d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 276bd3b97c97426daf802e1f57feb265 consensus state has no leader
I20251026 02:56:25.432587 2372 tablet_replica.cc:333] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:25.432744 2377 ts_tablet_manager.cc:1929] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.432781 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 0020cb1f190c49978c5322d387a47f71 (table default.Table_689f802c9e4b457eb71fb78898fe14b0 [id=05242557de9a4a19948b8591ce73ab28]) successfully deleted
I20251026 02:56:25.432886 2377 log.cc:1199] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/d8ae08997af74b66b7d6375212eee4c2
I20251026 02:56:25.433104 2372 raft_consensus.cc:2243] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:25.433235 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 340bf8c78a88413299b61a52f0a578f3 consensus state has no leader
I20251026 02:56:25.433386 2372 raft_consensus.cc:2272] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.433575 2377 ts_tablet_manager.cc:1950] T d8ae08997af74b66b7d6375212eee4c2 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:25.434505 2372 ts_tablet_manager.cc:1916] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.435200 2381 tablet_bootstrap.cc:654] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.440730 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet f6d6f024d8b943d5acf69b64bc7c7214 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a3491b303f94fe2a2c838297f73d464) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.443033 2381 tablet_bootstrap.cc:492] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.443233 2381 ts_tablet_manager.cc:1403] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.003s
I20251026 02:56:25.444387 2381 raft_consensus.cc:359] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.444638 2381 raft_consensus.cc:385] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.444749 2381 raft_consensus.cc:740] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.445056 2381 consensus_queue.cc:260] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.445756 2381 ts_tablet_manager.cc:1434] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:25.446164 2381 tablet_bootstrap.cc:492] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.447448 1656 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56] alter complete (version 2)
I20251026 02:56:25.452631 2372 ts_tablet_manager.cc:1929] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.452800 2372 log.cc:1199] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7176cbe8f64546b7871d047c0e0e5b2a
I20251026 02:56:25.453282 2372 ts_tablet_manager.cc:1950] T 7176cbe8f64546b7871d047c0e0e5b2a P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:25.454939 2117 raft_consensus.cc:1275] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:25.455472 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7176cbe8f64546b7871d047c0e0e5b2a (table default.Table_c46d7af7b60a48188efbea88a1b7a72b [id=93bef2a8d7b649e880042bfdbb1e08ac]) successfully deleted
I20251026 02:56:25.456498 2381 tablet_bootstrap.cc:654] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.456534 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4e5fa0edbee549878f707b605c7c0180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:25.456717 2648 tablet.cc:1722] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251026 02:56:25.457126 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e5fa0edbee549878f707b605c7c0180 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.458063 2372 tablet_replica.cc:333] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:25.458434 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5af1941b0082444c8bd348a080af20ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:25.458585 2372 raft_consensus.cc:2243] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.458802 2372 raft_consensus.cc:2272] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:25.458945 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5af1941b0082444c8bd348a080af20ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.460364 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ad4b4db01c7d4fe0a4e99f2f54cee0e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.460852 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad4b4db01c7d4fe0a4e99f2f54cee0e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.461205 2372 ts_tablet_manager.cc:1916] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.460389 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5ae1979aa25849fe8b780ecc617bb7e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.461908 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ae1979aa25849fe8b780ecc617bb7e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.464766 2643 consensus_queue.cc:1048] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:25.468493 2381 tablet_bootstrap.cc:492] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.468739 2381 ts_tablet_manager.cc:1403] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.004s
I20251026 02:56:25.469789 2381 raft_consensus.cc:359] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.470041 2381 raft_consensus.cc:385] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.470126 2381 raft_consensus.cc:740] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.470402 2381 consensus_queue.cc:260] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.471042 2381 ts_tablet_manager.cc:1434] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:25.471436 2381 tablet_bootstrap.cc:492] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.471766 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_03a9f618938a45aab881c91a801cc32c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.472410 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03a9f618938a45aab881c91a801cc32c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.474093 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_02a57a3ced604a78bd1d59268a599261"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:25.474282 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_194accfad8444dc788caa3b84dd315dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.474642 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02a57a3ced604a78bd1d59268a599261 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:25.475001 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_194accfad8444dc788caa3b84dd315dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.476356 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet d8ae08997af74b66b7d6375212eee4c2 (table default.Table_c095574955cf44d7aecfbca77de4ad1f [id=789c86929bf94d2ca3fb5289f598e292]) successfully deleted
W20251026 02:56:25.476701 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet b1a2522801364693be0bda6fa45a697e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1a2522801364693be0bda6fa45a697e already in progress: creating tablet
W20251026 02:56:25.476958 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 86d8e1e21c5f440d8baf76c357d01ad2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86d8e1e21c5f440d8baf76c357d01ad2 already in progress: creating tablet
W20251026 02:56:25.477200 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet f6d6f024d8b943d5acf69b64bc7c7214 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6d6f024d8b943d5acf69b64bc7c7214 already in progress: creating tablet
W20251026 02:56:25.477471 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 86d8e1e21c5f440d8baf76c357d01ad2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86d8e1e21c5f440d8baf76c357d01ad2 already in progress: creating tablet
W20251026 02:56:25.477687 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet f6d6f024d8b943d5acf69b64bc7c7214 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6d6f024d8b943d5acf69b64bc7c7214 already in progress: creating tablet
I20251026 02:56:25.478240 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_83bdf2cfef4042748553ebe8ae34996a" } new_table_name: "default.Table_620d0c4122484e579155fe90d2d5c491" modify_external_catalogs: true
I20251026 02:56:25.478655 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f12577c6a60b4858b2e778692827a023"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.479387 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f12577c6a60b4858b2e778692827a023 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.479483 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b78fbd41028a4314ae6a81090f004797"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.479944 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b78fbd41028a4314ae6a81090f004797 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.482635 2087 tablet_service.cc:1505] Processing CreateTablet for tablet be1c0948c0ae4563b7cb644521914cb9 (DEFAULT_TABLE table=default.Table_d197f56404544c8cb996d53edb9559a4 [id=777761bfdc67409bb98be7cf82bfa547]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.483260 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be1c0948c0ae4563b7cb644521914cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.485582 2375 tablet_bootstrap.cc:492] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.485730 2375 ts_tablet_manager.cc:1403] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.150s user 0.001s sys 0.009s
I20251026 02:56:25.486083 2084 tablet_service.cc:1505] Processing CreateTablet for tablet dba102838945453cbf44e1cfe505a52c (DEFAULT_TABLE table=default.Table_6fd0ab283f2c4b50a4e6cc4cb073c742 [id=92310ee99b5c428c9773b0f7a5462a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.486644 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba102838945453cbf44e1cfe505a52c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.486672 2375 raft_consensus.cc:359] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.486881 2375 raft_consensus.cc:385] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.486989 2375 raft_consensus.cc:740] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.487231 2375 consensus_queue.cc:260] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.488477 1979 raft_consensus.cc:3060] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.488761 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_65488c1f07e542b5973be5cfd122787c" } modify_external_catalogs: true
I20251026 02:56:25.488930 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_65488c1f07e542b5973be5cfd122787c" } modify_external_catalogs: true
I20251026 02:56:25.491588 1979 raft_consensus.cc:1275] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:25.493213 2440 consensus_queue.cc:1048] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:25.496726 2628 tablet.cc:1722] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:25.505998 1662 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 98fe44aab33b40eba819386c3bffca46 (table default.Table_481575c8f05747c2954c9c79a0ebde9c [id=91d789aff2464f73b60e03b52126a396]). Expected version 1 got 0
I20251026 02:56:25.506403 1662 catalog_manager.cc:5649] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:25.508020 1662 catalog_manager.cc:5649] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:25.508915 1662 catalog_manager.cc:5649] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:25.509361 1662 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 23fd1284c9cb41bd98209ccbfe342bca (table default.Table_83bdf2cfef4042748553ebe8ae34996a [id=6734e5731cc94e71b43c76512f649d57]). Expected version 1 got 0
I20251026 02:56:25.509627 1662 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 1d8cdf79a97149618bcaf0fefee1dfeb (table default.Table_424693529d0d4c829755adc2c9835c0e [id=690ec15818d34cdfa990c8fcd36a19cd]). Expected version 1 got 0
I20251026 02:56:25.512669 1960 tablet_service.cc:1505] Processing CreateTablet for tablet be1c0948c0ae4563b7cb644521914cb9 (DEFAULT_TABLE table=default.Table_d197f56404544c8cb996d53edb9559a4 [id=777761bfdc67409bb98be7cf82bfa547]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.513121 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be1c0948c0ae4563b7cb644521914cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.516655 2381 tablet_bootstrap.cc:654] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.521466 2235 tablet_service.cc:1505] Processing CreateTablet for tablet be1c0948c0ae4563b7cb644521914cb9 (DEFAULT_TABLE table=default.Table_d197f56404544c8cb996d53edb9559a4 [id=777761bfdc67409bb98be7cf82bfa547]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.521891 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be1c0948c0ae4563b7cb644521914cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.522989 2687 tablet.cc:1722] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:25.524722 2375 ts_tablet_manager.cc:1434] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20251026 02:56:25.525079 2375 tablet_bootstrap.cc:492] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.526719 2375 tablet_bootstrap.cc:654] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.530783 1960 tablet_service.cc:1505] Processing CreateTablet for tablet dba102838945453cbf44e1cfe505a52c (DEFAULT_TABLE table=default.Table_6fd0ab283f2c4b50a4e6cc4cb073c742 [id=92310ee99b5c428c9773b0f7a5462a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.531349 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba102838945453cbf44e1cfe505a52c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.531860 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 741792cf96164695b03de3fb86b12602 (DEFAULT_TABLE table=default.Table_d4304259ac684b8097cdbf5511ab67d7 [id=d64901838ed94621ad11692d9e0aa80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.532681 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 741792cf96164695b03de3fb86b12602. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.535907 2235 tablet_service.cc:1505] Processing CreateTablet for tablet dba102838945453cbf44e1cfe505a52c (DEFAULT_TABLE table=default.Table_6fd0ab283f2c4b50a4e6cc4cb073c742 [id=92310ee99b5c428c9773b0f7a5462a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.536541 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba102838945453cbf44e1cfe505a52c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.538196 2084 tablet_service.cc:1505] Processing CreateTablet for tablet c4b0933f71cb4ac7a58986d16b613e64 (DEFAULT_TABLE table=default.Table_3c56d6d5af8c44139ac28b8cc9945a9f [id=cba6d4517d244d7580e525729e871745]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.539175 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b0933f71cb4ac7a58986d16b613e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.539100 2100 tablet_service.cc:1505] Processing CreateTablet for tablet aa0aa6cdd0de4b0a91f553c2341e9c01 (DEFAULT_TABLE table=default.Table_1a48da091e984e09aaffb9caed79f2c1 [id=1b3988cec795459f8666476a9f979f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.539830 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa0aa6cdd0de4b0a91f553c2341e9c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.542414 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 741792cf96164695b03de3fb86b12602 (DEFAULT_TABLE table=default.Table_d4304259ac684b8097cdbf5511ab67d7 [id=d64901838ed94621ad11692d9e0aa80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.542816 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 741792cf96164695b03de3fb86b12602. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.544584 2375 tablet_bootstrap.cc:492] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.544754 2375 ts_tablet_manager.cc:1403] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.001s
I20251026 02:56:25.545552 2375 raft_consensus.cc:359] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.545635 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 8f3ae3a377af46449c3faca339f86053 (DEFAULT_TABLE table=default.Table_249435859b1e471585bf137537ac2446 [id=31f977e55ba041f59bcc721d469746fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.545753 2375 raft_consensus.cc:385] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.545840 2375 raft_consensus.cc:740] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.546118 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f3ae3a377af46449c3faca339f86053. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.546090 2375 consensus_queue.cc:260] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.549474 2088 tablet_service.cc:1505] Processing CreateTablet for tablet dbf2a1ce78154415b23ab6d0efbc1f52 (DEFAULT_TABLE table=default.Table_22ae79375e8e49d4b6d146d087ffa64b [id=4c170eaba0fd422f9fb0ce5e63e5c746]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.549958 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf2a1ce78154415b23ab6d0efbc1f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.554140 2378 tablet_bootstrap.cc:492] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:25.554387 2378 ts_tablet_manager.cc:1403] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.137s user 0.002s sys 0.001s
I20251026 02:56:25.555217 2378 raft_consensus.cc:359] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.555392 2378 raft_consensus.cc:385] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.555466 2378 raft_consensus.cc:740] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.555683 2378 consensus_queue.cc:260] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.556231 2378 ts_tablet_manager.cc:1434] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:25.556546 2378 tablet_bootstrap.cc:492] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:25.561084 2648 tablet.cc:1722] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:25.562624 2378 tablet_bootstrap.cc:654] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.568048 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 741792cf96164695b03de3fb86b12602 (DEFAULT_TABLE table=default.Table_d4304259ac684b8097cdbf5511ab67d7 [id=d64901838ed94621ad11692d9e0aa80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.568257 2375 ts_tablet_manager.cc:1434] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251026 02:56:25.568192 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 046a34f9408b406ba08471421f422064 (DEFAULT_TABLE table=default.Table_e2d286228a8244d2bded2ee992c5a064 [id=bc8a2d95ed134fae9f3e9deeac4f130a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.568576 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 741792cf96164695b03de3fb86b12602. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.568619 2375 tablet_bootstrap.cc:492] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.568708 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 046a34f9408b406ba08471421f422064. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.571715 2375 tablet_bootstrap.cc:654] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.572592 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 8688a0bf946841deb5d88c931538b129 (DEFAULT_TABLE table=default.Table_a6e8714ddccb4139adb6c8320e06500b [id=bb34b75c6fcb4ea6b201d05be4982a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.572985 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8688a0bf946841deb5d88c931538b129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.574215 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 3c3050e72d42417ab58a05f7e3aa56b3 (DEFAULT_TABLE table=default.Table_1fba8934d98d4fdb8b2c9a0b5d01e810 [id=b9b139445a9040398621353ad052f3c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.574572 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c3050e72d42417ab58a05f7e3aa56b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.575814 2375 tablet_bootstrap.cc:492] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.575980 2375 ts_tablet_manager.cc:1403] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251026 02:56:25.576903 2375 raft_consensus.cc:359] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.577168 2375 raft_consensus.cc:385] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.577283 2375 raft_consensus.cc:740] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.577586 2375 consensus_queue.cc:260] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.578318 2375 ts_tablet_manager.cc:1434] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251026 02:56:25.578678 2375 tablet_bootstrap.cc:492] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.586442 2643 raft_consensus.cc:493] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.586637 2643 raft_consensus.cc:515] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.587311 2643 leader_election.cc:290] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:25.589349 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c647782049243d6bcd595f1a10d89c6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:25.590572 2194 leader_election.cc:343] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:25.590793 2195 leader_election.cc:343] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 0c647782049243d6bcd595f1a10d89c6
I20251026 02:56:25.590896 2195 leader_election.cc:304] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:25.592478 2643 raft_consensus.cc:2749] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:25.596889 2375 tablet_bootstrap.cc:654] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.596817 1957 tablet_service.cc:1505] Processing CreateTablet for tablet c4b0933f71cb4ac7a58986d16b613e64 (DEFAULT_TABLE table=default.Table_3c56d6d5af8c44139ac28b8cc9945a9f [id=cba6d4517d244d7580e525729e871745]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.597280 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b0933f71cb4ac7a58986d16b613e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.606189 1953 tablet_service.cc:1505] Processing CreateTablet for tablet aa0aa6cdd0de4b0a91f553c2341e9c01 (DEFAULT_TABLE table=default.Table_1a48da091e984e09aaffb9caed79f2c1 [id=1b3988cec795459f8666476a9f979f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.606812 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa0aa6cdd0de4b0a91f553c2341e9c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.616370 2241 tablet_service.cc:1505] Processing CreateTablet for tablet c4b0933f71cb4ac7a58986d16b613e64 (DEFAULT_TABLE table=default.Table_3c56d6d5af8c44139ac28b8cc9945a9f [id=cba6d4517d244d7580e525729e871745]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.616947 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b0933f71cb4ac7a58986d16b613e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.625813 1951 tablet_service.cc:1505] Processing CreateTablet for tablet dbf2a1ce78154415b23ab6d0efbc1f52 (DEFAULT_TABLE table=default.Table_22ae79375e8e49d4b6d146d087ffa64b [id=4c170eaba0fd422f9fb0ce5e63e5c746]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.626338 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf2a1ce78154415b23ab6d0efbc1f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.630213 2372 ts_tablet_manager.cc:1929] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.630390 2372 log.cc:1199] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/cfed3523205548508d1101586ee3e2a9
I20251026 02:56:25.630800 2372 ts_tablet_manager.cc:1950] T cfed3523205548508d1101586ee3e2a9 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:25.626361 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 8f3ae3a377af46449c3faca339f86053 (DEFAULT_TABLE table=default.Table_249435859b1e471585bf137537ac2446 [id=31f977e55ba041f59bcc721d469746fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.634814 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f3ae3a377af46449c3faca339f86053. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.641775 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet cfed3523205548508d1101586ee3e2a9 (table default.Table_e4fbc44faf984982986abfb227d6663d [id=b35a8189bbc04e94bb097c9e13a38ec6]) successfully deleted
I20251026 02:56:25.645762 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 046a34f9408b406ba08471421f422064 (DEFAULT_TABLE table=default.Table_e2d286228a8244d2bded2ee992c5a064 [id=bc8a2d95ed134fae9f3e9deeac4f130a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.646342 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 046a34f9408b406ba08471421f422064. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.651662 2372 tablet_replica.cc:333] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:25.651916 2372 raft_consensus.cc:2243] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.652120 2372 raft_consensus.cc:2272] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.659098 2372 ts_tablet_manager.cc:1916] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:25.671728 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23fd1284c9cb41bd98209ccbfe342bca consensus state has no leader
I20251026 02:56:25.672463 2375 tablet_bootstrap.cc:492] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.672662 2375 ts_tablet_manager.cc:1403] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.094s user 0.018s sys 0.031s
I20251026 02:56:25.672927 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet a378e913112c443db869016e3b98fd0e from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:25.673408 2375 raft_consensus.cc:359] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.673620 2375 raft_consensus.cc:385] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.673686 2375 raft_consensus.cc:740] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.673974 2375 consensus_queue.cc:260] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.674355 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet 03da415f3a944a9a9679b3a45295ce35 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:25.674420 2375 ts_tablet_manager.cc:1434] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:25.674860 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7b08384413864da0a28224dd7201c263" } modify_external_catalogs: true
I20251026 02:56:25.675043 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7b08384413864da0a28224dd7201c263" } modify_external_catalogs: true
I20251026 02:56:25.676555 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_389946812a3d4337b46c0b2829ab935f" } new_table_name: "default.Table_fa634331e4bd4c14a102908bc41edba1" modify_external_catalogs: true
I20251026 02:56:25.678306 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4578a4ce81ea42cd878859a8167babff" } modify_external_catalogs: true
I20251026 02:56:25.678539 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4578a4ce81ea42cd878859a8167babff" } modify_external_catalogs: true
I20251026 02:56:25.680137 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4c63b6bed4b1437f9a11cba9c5d09d15" } new_table_name: "default.Table_21d71fb70373400396a75b6e52416078" modify_external_catalogs: true
I20251026 02:56:25.683133 2375 tablet_bootstrap.cc:492] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.682837 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 3c3050e72d42417ab58a05f7e3aa56b3 (DEFAULT_TABLE table=default.Table_1fba8934d98d4fdb8b2c9a0b5d01e810 [id=b9b139445a9040398621353ad052f3c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.683862 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c3050e72d42417ab58a05f7e3aa56b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.684603 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 8688a0bf946841deb5d88c931538b129 (DEFAULT_TABLE table=default.Table_a6e8714ddccb4139adb6c8320e06500b [id=bb34b75c6fcb4ea6b201d05be4982a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.685009 2375 tablet_bootstrap.cc:654] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.685240 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8688a0bf946841deb5d88c931538b129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.697362 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d2dcec579c684b92a92d98fb59ee3da1" } modify_external_catalogs: true
I20251026 02:56:25.697672 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d2dcec579c684b92a92d98fb59ee3da1" } modify_external_catalogs: true
I20251026 02:56:25.701278 2661 raft_consensus.cc:493] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.701510 2661 raft_consensus.cc:515] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.701910 2372 ts_tablet_manager.cc:1929] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.702049 2372 log.cc:1199] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/461412f3ad1e4186b1302cdf5a624ed4
I20251026 02:56:25.702275 2661 leader_election.cc:290] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:25.702461 2372 ts_tablet_manager.cc:1950] T 461412f3ad1e4186b1302cdf5a624ed4 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:25.703423 1916 leader_election.cc:343] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 7036a159e88b4171942dd238f8af3ed3
I20251026 02:56:25.703518 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 461412f3ad1e4186b1302cdf5a624ed4 (table default.Table_095bb281b8804e38b5bd56a1dbabacfa [id=6cab490620ef48f18ab808cd5d6c6af8]) successfully deleted
I20251026 02:56:25.706394 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet aa399e6407fd4c4d9871d3d5ccf894ec
I20251026 02:56:25.711126 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet aa399e6407fd4c4d9871d3d5ccf894ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:25.713203 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet f6d6f024d8b943d5acf69b64bc7c7214 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6d6f024d8b943d5acf69b64bc7c7214 already in progress: creating tablet
I20251026 02:56:25.713253 2378 tablet_bootstrap.cc:492] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:25.713454 2378 ts_tablet_manager.cc:1403] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.157s user 0.015s sys 0.032s
W20251026 02:56:25.713771 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet aa399e6407fd4c4d9871d3d5ccf894ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa399e6407fd4c4d9871d3d5ccf894ec already in progress: creating tablet
I20251026 02:56:25.714622 2378 raft_consensus.cc:359] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.714888 2378 raft_consensus.cc:385] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.714993 2378 raft_consensus.cc:740] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.715387 2378 consensus_queue.cc:260] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.720537 2378 ts_tablet_manager.cc:1434] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20251026 02:56:25.721127 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_03bc36a9fd184c28abd1caf6e5ec4f7a" } modify_external_catalogs: true
I20251026 02:56:25.721352 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_03bc36a9fd184c28abd1caf6e5ec4f7a" } modify_external_catalogs: true
I20251026 02:56:25.725189 2378 tablet_bootstrap.cc:492] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:25.727372 2378 tablet_bootstrap.cc:654] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.727648 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 3327d26a2c2c4d0b86d98151f5940bd7 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:25.735545 2237 tablet_service.cc:1505] Processing CreateTablet for tablet aa0aa6cdd0de4b0a91f553c2341e9c01 (DEFAULT_TABLE table=default.Table_1a48da091e984e09aaffb9caed79f2c1 [id=1b3988cec795459f8666476a9f979f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.736008 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa0aa6cdd0de4b0a91f553c2341e9c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.740484 2378 tablet_bootstrap.cc:492] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:25.740674 2378 ts_tablet_manager.cc:1403] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20251026 02:56:25.741614 2378 raft_consensus.cc:359] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.741873 2378 raft_consensus.cc:385] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.742002 2378 raft_consensus.cc:740] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.742409 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 8f3ae3a377af46449c3faca339f86053 (DEFAULT_TABLE table=default.Table_249435859b1e471585bf137537ac2446 [id=31f977e55ba041f59bcc721d469746fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.742992 2381 tablet_bootstrap.cc:492] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.743202 2381 ts_tablet_manager.cc:1403] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.272s user 0.018s sys 0.015s
I20251026 02:56:25.743505 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet f8533f1c191042ee9f2c0b99972fe931 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:25.744040 2230 tablet_service.cc:1505] Processing CreateTablet for tablet dbf2a1ce78154415b23ab6d0efbc1f52 (DEFAULT_TABLE table=default.Table_22ae79375e8e49d4b6d146d087ffa64b [id=4c170eaba0fd422f9fb0ce5e63e5c746]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.744129 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 98fe44aab33b40eba819386c3bffca46 (table default.Table_481575c8f05747c2954c9c79a0ebde9c [id=91d789aff2464f73b60e03b52126a396]). Expected version 1 got 0
I20251026 02:56:25.744246 2381 raft_consensus.cc:359] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.744444 2381 raft_consensus.cc:385] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.744388 2378 consensus_queue.cc:260] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.744532 2381 raft_consensus.cc:740] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.744627 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 8688a0bf946841deb5d88c931538b129 (DEFAULT_TABLE table=default.Table_a6e8714ddccb4139adb6c8320e06500b [id=bb34b75c6fcb4ea6b201d05be4982a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.744834 2381 consensus_queue.cc:260] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.745189 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 3c3050e72d42417ab58a05f7e3aa56b3 (DEFAULT_TABLE table=default.Table_1fba8934d98d4fdb8b2c9a0b5d01e810 [id=b9b139445a9040398621353ad052f3c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.745361 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 046a34f9408b406ba08471421f422064 (DEFAULT_TABLE table=default.Table_e2d286228a8244d2bded2ee992c5a064 [id=bc8a2d95ed134fae9f3e9deeac4f130a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.745479 2381 ts_tablet_manager.cc:1434] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20251026 02:56:25.746094 1915 leader_election.cc:343] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 7036a159e88b4171942dd238f8af3ed3
I20251026 02:56:25.746192 1654 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 98fe44aab33b40eba819386c3bffca46 (table default.Table_481575c8f05747c2954c9c79a0ebde9c [id=91d789aff2464f73b60e03b52126a396]). Expected version 1 got 0
I20251026 02:56:25.746233 1915 leader_election.cc:304] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:25.747085 2661 raft_consensus.cc:2749] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:25.747656 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f3ae3a377af46449c3faca339f86053. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.748499 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 046a34f9408b406ba08471421f422064. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.749517 2381 tablet_bootstrap.cc:492] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.749975 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c3050e72d42417ab58a05f7e3aa56b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.751837 2381 tablet_bootstrap.cc:654] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.752553 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8688a0bf946841deb5d88c931538b129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.753355 2378 ts_tablet_manager.cc:1434] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251026 02:56:25.756664 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf2a1ce78154415b23ab6d0efbc1f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.760571 2381 tablet_bootstrap.cc:492] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.760641 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_acad6a9e93e24617aff27d46e9c5c099" } new_table_name: "default.Table_e9da6904da2c4d15bb17df7e9bbb3221" modify_external_catalogs: true
I20251026 02:56:25.760748 2381 ts_tablet_manager.cc:1403] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20251026 02:56:25.761660 2381 raft_consensus.cc:359] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.761876 2381 raft_consensus.cc:385] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.761981 2381 raft_consensus.cc:740] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.762092 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_18e8723148dd46e6a56e5716a15d01bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:25.762264 2381 consensus_queue.cc:260] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:25.762684 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18e8723148dd46e6a56e5716a15d01bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.763288 2378 tablet_bootstrap.cc:492] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:25.766537 1662 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (table default.Table_389946812a3d4337b46c0b2829ab935f [id=1d09f5b9fa164ac783f0be9332eac4c3]). Expected version 3 got 0
I20251026 02:56:25.767234 2381 ts_tablet_manager.cc:1434] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251026 02:56:25.768689 2378 tablet_bootstrap.cc:654] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.770438 2381 tablet_bootstrap.cc:492] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.773952 2381 tablet_bootstrap.cc:654] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.775967 2378 tablet_bootstrap.cc:492] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
W20251026 02:56:25.778681 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 740f7c7f186f45ceaa4c40e76419e3a1 consensus state has no leader
I20251026 02:56:25.776172 2378 ts_tablet_manager.cc:1403] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20251026 02:56:25.781375 2378 raft_consensus.cc:359] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.781657 2378 raft_consensus.cc:385] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.781740 2378 raft_consensus.cc:740] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.783466 2378 consensus_queue.cc:260] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.787304 2378 ts_tablet_manager.cc:1434] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251026 02:56:25.788720 2381 tablet_bootstrap.cc:492] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.788882 2381 ts_tablet_manager.cc:1403] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.002s
I20251026 02:56:25.788779 2378 tablet_bootstrap.cc:492] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:25.790982 2375 tablet_bootstrap.cc:492] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.791159 2375 ts_tablet_manager.cc:1403] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.108s user 0.021s sys 0.018s
I20251026 02:56:25.791956 2375 raft_consensus.cc:359] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.792133 2375 raft_consensus.cc:385] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.792253 2375 raft_consensus.cc:740] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.792536 2375 consensus_queue.cc:260] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.793202 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9156314f98ac4ed9981f6f13fc585043
I20251026 02:56:25.794145 2375 ts_tablet_manager.cc:1434] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251026 02:56:25.794335 2378 tablet_bootstrap.cc:654] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.794461 2375 tablet_bootstrap.cc:492] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:25.795799 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 9156314f98ac4ed9981f6f13fc585043 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:25.796476 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9156314f98ac4ed9981f6f13fc585043 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9156314f98ac4ed9981f6f13fc585043 already in progress: creating tablet
I20251026 02:56:25.789764 2381 raft_consensus.cc:359] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.796818 2381 raft_consensus.cc:385] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.797195 2375 tablet_bootstrap.cc:654] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.797327 2381 raft_consensus.cc:740] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.797610 2381 consensus_queue.cc:260] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.801762 2381 ts_tablet_manager.cc:1434] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251026 02:56:25.802495 2381 tablet_bootstrap.cc:492] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.804651 2381 tablet_bootstrap.cc:654] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:25.807794 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca4611c0a494425ca50d599b310872c8 consensus state has no leader
I20251026 02:56:25.811224 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d300a4e488f043afa43e544db231cc02" } new_table_name: "default.Table_ffa4b05d8c84446297b336b55bbf5c6d" modify_external_catalogs: true
I20251026 02:56:25.813213 2381 tablet_bootstrap.cc:492] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.813388 2381 ts_tablet_manager.cc:1403] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20251026 02:56:25.813685 2378 tablet_bootstrap.cc:492] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:25.813839 2378 ts_tablet_manager.cc:1403] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.001s
I20251026 02:56:25.814164 2381 raft_consensus.cc:359] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.814374 2381 raft_consensus.cc:385] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.814445 2381 raft_consensus.cc:740] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.814652 2378 raft_consensus.cc:359] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.814882 2378 raft_consensus.cc:385] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.815012 2378 raft_consensus.cc:740] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.815308 2378 consensus_queue.cc:260] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.816504 2381 consensus_queue.cc:260] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.818766 2381 ts_tablet_manager.cc:1434] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251026 02:56:25.819707 2381 tablet_bootstrap.cc:492] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.821076 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 634d913c98c842a49d97a54f23752d26
I20251026 02:56:25.821915 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 8dbc12d49b094dbeadad2e76916ba9af from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:25.823513 2381 tablet_bootstrap.cc:654] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.823480 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 634d913c98c842a49d97a54f23752d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:25.824949 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 634d913c98c842a49d97a54f23752d26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 634d913c98c842a49d97a54f23752d26 already in progress: creating tablet
I20251026 02:56:25.826663 2378 ts_tablet_manager.cc:1434] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251026 02:56:25.828603 2378 tablet_bootstrap.cc:492] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:25.831457 2381 tablet_bootstrap.cc:492] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.832149 2381 ts_tablet_manager.cc:1403] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20251026 02:56:25.832722 2378 tablet_bootstrap.cc:654] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.833232 2381 raft_consensus.cc:359] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.833464 2381 raft_consensus.cc:385] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.833571 2381 raft_consensus.cc:740] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.835393 2381 consensus_queue.cc:260] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.837821 2381 ts_tablet_manager.cc:1434] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251026 02:56:25.838963 2381 tablet_bootstrap.cc:492] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.842875 2381 tablet_bootstrap.cc:654] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.842849 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 863ce9e1adb444b7b1399b968ea79bb4
I20251026 02:56:25.844233 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 863ce9e1adb444b7b1399b968ea79bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:25.845278 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 863ce9e1adb444b7b1399b968ea79bb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 863ce9e1adb444b7b1399b968ea79bb4 already in progress: creating tablet
I20251026 02:56:25.846976 2378 tablet_bootstrap.cc:492] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:25.847136 2378 ts_tablet_manager.cc:1403] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251026 02:56:25.847848 2378 raft_consensus.cc:359] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.848042 2378 raft_consensus.cc:385] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.848107 2378 raft_consensus.cc:740] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.848372 2378 consensus_queue.cc:260] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.848881 2378 ts_tablet_manager.cc:1434] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251026 02:56:25.850299 1647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.60:33729, user_credentials={real_user=slave}} blocked reactor thread for 173544us
I20251026 02:56:25.852641 2381 tablet_bootstrap.cc:492] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.853531 2381 ts_tablet_manager.cc:1403] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20251026 02:56:25.853664 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet aa399e6407fd4c4d9871d3d5ccf894ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.854282 2381 raft_consensus.cc:359] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.854357 2661 raft_consensus.cc:493] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.854460 2381 raft_consensus.cc:385] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.854517 2381 raft_consensus.cc:740] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.854518 2661 raft_consensus.cc:515] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.854885 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3327d26a2c2c4d0b86d98151f5940bd7
I20251026 02:56:25.855144 2661 leader_election.cc:290] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:25.855417 1661 catalog_manager.cc:6719] ReplaceTablet: tablet 3327d26a2c2c4d0b86d98151f5940bd7 deleted and replaced by tablet 7177931353164b2aa82c00b2807ccd63
I20251026 02:56:25.855996 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 3327d26a2c2c4d0b86d98151f5940bd7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7177931353164b2aa82c00b2807ccd63) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.856938 2378 tablet_bootstrap.cc:492] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:25.859094 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3327d26a2c2c4d0b86d98151f5940bd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3327d26a2c2c4d0b86d98151f5940bd7 already in progress: creating tablet
I20251026 02:56:25.859691 2381 consensus_queue.cc:260] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.859987 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet aa399e6407fd4c4d9871d3d5ccf894ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.860994 2381 ts_tablet_manager.cc:1434] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251026 02:56:25.861344 2381 tablet_bootstrap.cc:492] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:25.862026 1916 leader_election.cc:343] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 1c1596dd4394419a8bb6ea6a84be9fea
W20251026 02:56:25.863692 1915 leader_election.cc:343] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 1c1596dd4394419a8bb6ea6a84be9fea
I20251026 02:56:25.863842 1915 leader_election.cc:304] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:25.864121 2378 tablet_bootstrap.cc:654] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.864446 2661 raft_consensus.cc:2749] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:25.868059 2381 tablet_bootstrap.cc:654] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.870076 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 7439c051ca634fbe8b511d47ccede2ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.870354 2377 tablet_replica.cc:333] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:25.870564 2377 raft_consensus.cc:2243] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.870728 2377 raft_consensus.cc:2272] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:25.871115 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e62064fbdffb4954a3a1227efd6327ee consensus state has no leader
I20251026 02:56:25.871291 2377 ts_tablet_manager.cc:1916] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.875106 2643 raft_consensus.cc:493] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.875311 2643 raft_consensus.cc:515] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.875913 2643 leader_election.cc:290] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:25.877454 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7176cbe8f64546b7871d047c0e0e5b2a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:25.878432 2378 tablet_bootstrap.cc:492] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:25.878614 2378 ts_tablet_manager.cc:1403] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
W20251026 02:56:25.878780 2194 leader_election.cc:343] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:25.879280 2195 leader_election.cc:343] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 7176cbe8f64546b7871d047c0e0e5b2a
I20251026 02:56:25.879410 2195 leader_election.cc:304] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:25.879590 2378 raft_consensus.cc:359] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.879797 2378 raft_consensus.cc:385] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.879895 2378 raft_consensus.cc:740] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.879927 2643 raft_consensus.cc:2749] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:25.880144 2378 consensus_queue.cc:260] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.885129 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 634d913c98c842a49d97a54f23752d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.886057 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 863ce9e1adb444b7b1399b968ea79bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.886529 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 9156314f98ac4ed9981f6f13fc585043 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.887189 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1fd41bf4714546fba6cf85660cdf272d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.887703 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fd41bf4714546fba6cf85660cdf272d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.889350 2375 tablet_bootstrap.cc:492] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:25.889526 2375 ts_tablet_manager.cc:1403] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.095s user 0.020s sys 0.019s
I20251026 02:56:25.890084 2646 raft_consensus.cc:493] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.890251 2646 raft_consensus.cc:515] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.890275 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2dfe8bcc7f1e4933a4174bdab19cffd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.890820 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2dfe8bcc7f1e4933a4174bdab19cffd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.890839 2646 leader_election.cc:290] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:25.891582 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5580ac28a98c4ec4a1d83fb9f9bf25ee" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:25.892127 2053 leader_election.cc:343] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:25.893249 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet aa399e6407fd4c4d9871d3d5ccf894ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa399e6407fd4c4d9871d3d5ccf894ec already in progress: creating tablet
I20251026 02:56:25.893262 2375 raft_consensus.cc:359] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.893455 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d6690e6bf67454398637d9908b9c652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.893538 2375 raft_consensus.cc:385] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.893637 2375 raft_consensus.cc:740] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.893728 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ae08997af74b66b7d6375212eee4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.893903 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c647782049243d6bcd595f1a10d89c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 5caeb11322fe48348ebb523967152187) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.894001 2375 consensus_queue.cc:260] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:25.895207 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 634d913c98c842a49d97a54f23752d26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 634d913c98c842a49d97a54f23752d26 already in progress: creating tablet
I20251026 02:56:25.895498 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5580ac28a98c4ec4a1d83fb9f9bf25ee" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:25.896320 2053 leader_election.cc:343] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.896399 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 863ce9e1adb444b7b1399b968ea79bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.896492 2053 leader_election.cc:304] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:25.896590 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 9156314f98ac4ed9981f6f13fc585043 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.898430 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 634d913c98c842a49d97a54f23752d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.899093 1979 raft_consensus.cc:3060] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:25.899228 2117 raft_consensus.cc:1275] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:25.899808 2440 consensus_queue.cc:1048] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:25.902640 1979 raft_consensus.cc:1275] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:25.903088 2377 ts_tablet_manager.cc:1929] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.903226 2377 log.cc:1199] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7439c051ca634fbe8b511d47ccede2ff
I20251026 02:56:25.903620 2377 ts_tablet_manager.cc:1950] T 7439c051ca634fbe8b511d47ccede2ff P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:25.904266 2646 raft_consensus.cc:2749] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:25.905356 2440 consensus_queue.cc:1048] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:25.906970 2377 tablet_replica.cc:333] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:25.907437 2377 raft_consensus.cc:2243] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.908394 2646 raft_consensus.cc:493] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.908578 2646 raft_consensus.cc:515] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.907889 2377 raft_consensus.cc:2272] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.909163 2646 leader_election.cc:290] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:25.909816 2377 ts_tablet_manager.cc:1916] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.910559 2440 raft_consensus.cc:493] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.910555 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1b6257a3c8c4e1a8144ea45ff2a2a96" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:25.910813 2440 raft_consensus.cc:515] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.911486 2440 leader_election.cc:290] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:25.912142 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20dec74440cf4bb9949c462ec51e9889" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:25.912793 2194 leader_election.cc:343] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:25.913403 2053 leader_election.cc:343] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.913720 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1b6257a3c8c4e1a8144ea45ff2a2a96" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:25.913931 2195 leader_election.cc:343] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 20dec74440cf4bb9949c462ec51e9889
I20251026 02:56:25.914076 2195 leader_election.cc:304] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:25.914494 2053 leader_election.cc:343] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:25.914506 2440 raft_consensus.cc:2749] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:25.914615 2053 leader_election.cc:304] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:25.915218 2646 raft_consensus.cc:2749] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:25.915940 2087 tablet_service.cc:1505] Processing CreateTablet for tablet d65b0d79e2e54af9810022b683c3288b (DEFAULT_TABLE table=default.Table_194accfad8444dc788caa3b84dd315dd [id=30a5783033cb4f7f929e766fb47d92ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.885164 2378 ts_tablet_manager.cc:1434] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251026 02:56:25.916392 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d65b0d79e2e54af9810022b683c3288b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.916466 2378 tablet_bootstrap.cc:492] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:25.918354 2378 tablet_bootstrap.cc:654] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.921057 2084 tablet_service.cc:1505] Processing CreateTablet for tablet e9cd10092aac40068a1c7f85e514fd37 (DEFAULT_TABLE table=default.Table_02a57a3ced604a78bd1d59268a599261 [id=de5272bcc29644208ddd95bdec21254e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.921464 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9cd10092aac40068a1c7f85e514fd37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.924705 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 3b7d85a2941545ceaa16b22e8328b4ff (DEFAULT_TABLE table=default.Table_5ae1979aa25849fe8b780ecc617bb7e8 [id=9f6ac6400247491485d6bef30bf6cf4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.925209 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7d85a2941545ceaa16b22e8328b4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.897477 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4f8511a57eb54e5eaf42a6361676ff21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.929688 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 863ce9e1adb444b7b1399b968ea79bb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 863ce9e1adb444b7b1399b968ea79bb4 already in progress: creating tablet
W20251026 02:56:25.929965 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 9156314f98ac4ed9981f6f13fc585043 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9156314f98ac4ed9981f6f13fc585043 already in progress: creating tablet
I20251026 02:56:25.898550 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ec68beb934274468a4f028b3dbaf2cc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.901232 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85f9fcae83a04c6e907c7e9c8cd95dbe has no consensus state
I20251026 02:56:25.930559 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 3327d26a2c2c4d0b86d98151f5940bd7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7177931353164b2aa82c00b2807ccd63) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:25.929563 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 6108219254bd448c852096c0c967384b (DEFAULT_TABLE table=default.Table_5af1941b0082444c8bd348a080af20ee [id=6342ae7b8acf4137a16ebf978ce7e15b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.932885 2096 tablet_service.cc:1505] Processing CreateTablet for tablet ae95654d129441e08fcb3f3b355ffc22 (DEFAULT_TABLE table=default.Table_4e5fa0edbee549878f707b605c7c0180 [id=b0104213af32435589ae19109b7aee7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.933676 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 3327d26a2c2c4d0b86d98151f5940bd7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7177931353164b2aa82c00b2807ccd63) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:25.911932 1659 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 1d8cdf79a97149618bcaf0fefee1dfeb (table default.Table_424693529d0d4c829755adc2c9835c0e [id=690ec15818d34cdfa990c8fcd36a19cd]). Expected version 1 got 0
I20251026 02:56:25.935799 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae95654d129441e08fcb3f3b355ffc22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.941478 1662 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_481575c8f05747c2954c9c79a0ebde9c [id=91d789aff2464f73b60e03b52126a396] alter complete (version 1)
I20251026 02:56:25.944258 1957 tablet_service.cc:1505] Processing CreateTablet for tablet d65b0d79e2e54af9810022b683c3288b (DEFAULT_TABLE table=default.Table_194accfad8444dc788caa3b84dd315dd [id=30a5783033cb4f7f929e766fb47d92ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.944775 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d65b0d79e2e54af9810022b683c3288b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.944772 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1d7ffdaeec2240dcbe4cafdb96073a7d
W20251026 02:56:25.945685 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f8511a57eb54e5eaf42a6361676ff21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.946915 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6108219254bd448c852096c0c967384b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.949260 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 with delete_type TABLET_DATA_DELETED (replaced by tablet 94886ad6d6bb45f39261725169774ac9) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.949396 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet ac19c5a847454a23ab16831198ff2a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.949811 2372 tablet_replica.cc:333] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:25.950047 2372 raft_consensus.cc:2243] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.950254 2372 raft_consensus.cc:2272] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.951491 2372 ts_tablet_manager.cc:1916] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:25.952760 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:25.931833 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fc5e859548014044852b49ab2b64bd13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.953955 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc5e859548014044852b49ab2b64bd13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:25.955406 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec68beb934274468a4f028b3dbaf2cc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.957218 2381 tablet_bootstrap.cc:492] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:25.957410 2381 ts_tablet_manager.cc:1403] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.005s
I20251026 02:56:25.958199 2381 raft_consensus.cc:359] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.958376 2381 raft_consensus.cc:385] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:25.958452 2381 raft_consensus.cc:740] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:25.958666 2381 consensus_queue.cc:260] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.959105 2230 tablet_service.cc:1505] Processing CreateTablet for tablet d65b0d79e2e54af9810022b683c3288b (DEFAULT_TABLE table=default.Table_194accfad8444dc788caa3b84dd315dd [id=30a5783033cb4f7f929e766fb47d92ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:25.959511 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d65b0d79e2e54af9810022b683c3288b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:25.926704 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_85c523c971b34e68bed4afa5d1aac450"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.963166 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85c523c971b34e68bed4afa5d1aac450 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.898177 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6281fc8054c54d01b0376df4725d1adc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:25.964727 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6281fc8054c54d01b0376df4725d1adc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:25.965478 2372 ts_tablet_manager.cc:1929] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.965631 2372 log.cc:1199] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/c1b6257a3c8c4e1a8144ea45ff2a2a96
I20251026 02:56:25.966005 2372 ts_tablet_manager.cc:1950] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:25.967063 2372 tablet_replica.cc:333] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:25.967276 2372 raft_consensus.cc:2243] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:25.967466 2372 raft_consensus.cc:2272] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:25.969806 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 (table default.Table_d23ac2c9657b4adeaac9545d5bc9ae41 [id=9e897d33e6374ad4bc3c8acaae512645]) successfully deleted
I20251026 02:56:25.971685 2646 consensus_queue.cc:1048] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:25.975716 2381 ts_tablet_manager.cc:1434] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20251026 02:56:25.977734 2381 tablet_bootstrap.cc:492] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:25.984411 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 8667ffa25f2b478a987826d486400bbe (table default.Table_4fd5f39751624015ad57dd7fc77ce18a [id=f7f1e852a46f4129b2709fd58a6be034]). Expected version 1 got 0
I20251026 02:56:25.984763 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]). Expected version 1 got 0
I20251026 02:56:25.988262 2377 ts_tablet_manager.cc:1929] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:25.988878 2377 log.cc:1199] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3d6690e6bf67454398637d9908b9c652
I20251026 02:56:25.989364 2646 consensus_queue.cc:1048] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251026 02:56:25.989954 2646 raft_consensus.cc:493] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:25.993817 2381 tablet_bootstrap.cc:654] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:25.993762 2377 ts_tablet_manager.cc:1950] T 3d6690e6bf67454398637d9908b9c652 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:25.996876 2646 raft_consensus.cc:515] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:25.998394 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7dec83bfb9e946db90c10d526af2dfca" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:25.999244 2053 leader_election.cc:343] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.001437 2378 tablet_bootstrap.cc:492] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:26.001618 2378 ts_tablet_manager.cc:1403] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.085s user 0.000s sys 0.003s
I20251026 02:56:26.002111 2646 leader_election.cc:290] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.002487 2378 raft_consensus.cc:359] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.002733 2378 raft_consensus.cc:385] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.002807 2378 raft_consensus.cc:740] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.002919 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7dec83bfb9e946db90c10d526af2dfca" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:26.003055 2378 consensus_queue.cc:260] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.003603 2378 ts_tablet_manager.cc:1434] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251026 02:56:26.003705 2053 leader_election.cc:343] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.003850 2053 leader_election.cc:304] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.004371 2646 raft_consensus.cc:2749] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.006239 2377 tablet_replica.cc:333] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:26.007611 2377 raft_consensus.cc:2243] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.008971 2377 raft_consensus.cc:2272] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.013532 2377 ts_tablet_manager.cc:1916] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.016273 2375 ts_tablet_manager.cc:1434] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.127s user 0.001s sys 0.002s
I20251026 02:56:26.016678 2375 tablet_bootstrap.cc:492] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.019547 2661 raft_consensus.cc:493] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.019776 2661 raft_consensus.cc:515] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.020676 2661 leader_election.cc:290] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:26.021911 1916 leader_election.cc:343] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 3f60bcc5e4cb4fd19d2a5713f69814cd
I20251026 02:56:26.021968 2372 ts_tablet_manager.cc:1916] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.031575 2378 tablet_bootstrap.cc:492] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:26.033478 2378 tablet_bootstrap.cc:654] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.035732 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 3b7d85a2941545ceaa16b22e8328b4ff (DEFAULT_TABLE table=default.Table_5ae1979aa25849fe8b780ecc617bb7e8 [id=9f6ac6400247491485d6bef30bf6cf4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.036266 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7d85a2941545ceaa16b22e8328b4ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:26.038157 1915 leader_election.cc:343] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 3f60bcc5e4cb4fd19d2a5713f69814cd
I20251026 02:56:26.038214 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 6a3491b303f94fe2a2c838297f73d464 (DEFAULT_TABLE table=default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.038321 1915 leader_election.cc:304] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:26.038748 2661 raft_consensus.cc:2749] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.038725 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a3491b303f94fe2a2c838297f73d464. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.040302 1654 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 23fd1284c9cb41bd98209ccbfe342bca (table default.Table_620d0c4122484e579155fe90d2d5c491 [id=6734e5731cc94e71b43c76512f649d57]). Expected version 2 got 0
I20251026 02:56:26.041667 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe (DEFAULT_TABLE table=default.Table_ffa4b05d8c84446297b336b55bbf5c6d [id=87349d9c6c764382b828e716e7854c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.042625 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.044076 2230 tablet_service.cc:1505] Processing CreateTablet for tablet e9cd10092aac40068a1c7f85e514fd37 (DEFAULT_TABLE table=default.Table_02a57a3ced604a78bd1d59268a599261 [id=de5272bcc29644208ddd95bdec21254e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.044930 1957 tablet_service.cc:1505] Processing CreateTablet for tablet e9cd10092aac40068a1c7f85e514fd37 (DEFAULT_TABLE table=default.Table_02a57a3ced604a78bd1d59268a599261 [id=de5272bcc29644208ddd95bdec21254e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.045416 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9cd10092aac40068a1c7f85e514fd37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.045598 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9cd10092aac40068a1c7f85e514fd37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.047537 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dec83bfb9e946db90c10d526af2dfca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:26.048812 2755 tablet.cc:1722] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:26.047011 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee with delete_type TABLET_DATA_DELETED (replaced by tablet 634d913c98c842a49d97a54f23752d26) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:26.053505 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 23fd1284c9cb41bd98209ccbfe342bca (table default.Table_620d0c4122484e579155fe90d2d5c491 [id=6734e5731cc94e71b43c76512f649d57]). Expected version 2 got 0
I20251026 02:56:26.055776 2375 tablet_bootstrap.cc:654] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.058533 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 6108219254bd448c852096c0c967384b (DEFAULT_TABLE table=default.Table_5af1941b0082444c8bd348a080af20ee [id=6342ae7b8acf4137a16ebf978ce7e15b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.059099 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6108219254bd448c852096c0c967384b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.050001 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 3b7d85a2941545ceaa16b22e8328b4ff (DEFAULT_TABLE table=default.Table_5ae1979aa25849fe8b780ecc617bb7e8 [id=9f6ac6400247491485d6bef30bf6cf4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.051151 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 6108219254bd448c852096c0c967384b (DEFAULT_TABLE table=default.Table_5af1941b0082444c8bd348a080af20ee [id=6342ae7b8acf4137a16ebf978ce7e15b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.056715 2440 raft_consensus.cc:493] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.065080 2440 raft_consensus.cc:515] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.066251 2440 leader_election.cc:290] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
W20251026 02:56:26.066195 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet aa399e6407fd4c4d9871d3d5ccf894ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa399e6407fd4c4d9871d3d5ccf894ec already in progress: creating tablet
I20251026 02:56:26.066635 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7439c051ca634fbe8b511d47ccede2ff (table default.Table_be7d5ddd5e4e4b15862de09dc573f7f9 [id=354a2ac9115e4f2385537f1c16992882]) successfully deleted
I20251026 02:56:26.067214 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3d6690e6bf67454398637d9908b9c652 (table default.Table_e071f7a8a409464d810f12e647a4ff0e [id=c98c6fb4910449b0ad4513fc1c8f6f36]) successfully deleted
I20251026 02:56:26.067236 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0020cb1f190c49978c5322d387a47f71" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:26.067600 2195 leader_election.cc:343] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 0020cb1f190c49978c5322d387a47f71
W20251026 02:56:26.067737 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet d8ae08997af74b66b7d6375212eee4c2 because the tablet was not found. No further retry: Not found: Tablet not found: d8ae08997af74b66b7d6375212eee4c2
W20251026 02:56:26.067751 2194 leader_election.cc:343] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.067903 2194 leader_election.cc:304] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.068202 2440 raft_consensus.cc:2749] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.069468 1951 tablet_service.cc:1505] Processing CreateTablet for tablet ae95654d129441e08fcb3f3b355ffc22 (DEFAULT_TABLE table=default.Table_4e5fa0edbee549878f707b605c7c0180 [id=b0104213af32435589ae19109b7aee7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.070013 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae95654d129441e08fcb3f3b355ffc22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.076120 2381 tablet_bootstrap.cc:492] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.080138 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7d85a2941545ceaa16b22e8328b4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.080310 2381 ts_tablet_manager.cc:1403] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.104s user 0.008s sys 0.000s
I20251026 02:56:26.080736 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6108219254bd448c852096c0c967384b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.081138 2381 raft_consensus.cc:359] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.081351 2381 raft_consensus.cc:385] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.081426 2381 raft_consensus.cc:740] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.081667 2381 consensus_queue.cc:260] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.082218 2381 ts_tablet_manager.cc:1434] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:26.090703 2243 tablet_service.cc:1505] Processing CreateTablet for tablet ae95654d129441e08fcb3f3b355ffc22 (DEFAULT_TABLE table=default.Table_4e5fa0edbee549878f707b605c7c0180 [id=b0104213af32435589ae19109b7aee7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.090862 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.091282 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae95654d129441e08fcb3f3b355ffc22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.093376 2381 tablet_bootstrap.cc:492] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.097262 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 20dec74440cf4bb9949c462ec51e9889 with delete_type TABLET_DATA_DELETED (replaced by tablet c570e5f9a93c45bda32ec92c68794714) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.098217 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 7176cbe8f64546b7871d047c0e0e5b2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.099381 2381 tablet_bootstrap.cc:654] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.099526 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet ea910e98d17340efb816a8aa90f0ba7e with delete_type TABLET_DATA_DELETED (replaced by tablet 1370b9ad917344c093e5da3d0e9b1cf0) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.103988 2761 tablet.cc:1722] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:26.112746 2440 raft_consensus.cc:493] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.113060 2440 raft_consensus.cc:515] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.113158 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7488ff00e4c04293b85e265c4d686fc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.113721 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7488ff00e4c04293b85e265c4d686fc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.113741 2440 leader_election.cc:290] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:26.115072 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_03a9f618938a45aab881c91a801cc32c" } modify_external_catalogs: true
I20251026 02:56:26.115296 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_03a9f618938a45aab881c91a801cc32c" } modify_external_catalogs: true
I20251026 02:56:26.117712 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 7801f073d5d94076af9299492f86fad2 (DEFAULT_TABLE table=default.Table_03a9f618938a45aab881c91a801cc32c [id=12b41b47b5d243188eb73bd83a83bbdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.118305 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7801f073d5d94076af9299492f86fad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.119735 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d8cdf79a97149618bcaf0fefee1dfeb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:26.120038 2118 raft_consensus.cc:2468] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:26.120252 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_42c4c5e055054536b2b541711288a481"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:26.120437 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 0020cb1f190c49978c5322d387a47f71 with delete_type TABLET_DATA_DELETED (replaced by tablet 86d8e1e21c5f440d8baf76c357d01ad2) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:26.120828 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42c4c5e055054536b2b541711288a481 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.120862 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet cfed3523205548508d1101586ee3e2a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 44af74033ccf4d048475f12d164938f5) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.121644 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe (DEFAULT_TABLE table=default.Table_ffa4b05d8c84446297b336b55bbf5c6d [id=87349d9c6c764382b828e716e7854c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.121965 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_249435859b1e471585bf137537ac2446" } new_table_name: "default.Table_6a9abf3b541f42cda3bbf2f44d86c970" modify_external_catalogs: true
I20251026 02:56:26.122067 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.123219 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_194accfad8444dc788caa3b84dd315dd" } new_table_name: "default.Table_bcac335a2c9a479fb5dea86b2aeffde6" modify_external_catalogs: true
I20251026 02:56:26.123286 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c4bf189953554cb8854613da705a2b42" } modify_external_catalogs: true
I20251026 02:56:26.123442 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c4bf189953554cb8854613da705a2b42" } modify_external_catalogs: true
I20251026 02:56:26.124464 2195 leader_election.cc:304] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:26.124802 2440 raft_consensus.cc:2804] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:26.124912 2440 raft_consensus.cc:493] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:26.125021 2440 raft_consensus.cc:3060] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.125604 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4fd5f39751624015ad57dd7fc77ce18a" } modify_external_catalogs: true
I20251026 02:56:26.125792 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4fd5f39751624015ad57dd7fc77ce18a" } modify_external_catalogs: true
I20251026 02:56:26.126473 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1fd41bf4714546fba6cf85660cdf272d" } modify_external_catalogs: true
I20251026 02:56:26.126636 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1fd41bf4714546fba6cf85660cdf272d" } modify_external_catalogs: true
I20251026 02:56:26.127981 2440 raft_consensus.cc:515] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.128813 2440 leader_election.cc:290] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:26.133055 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d8cdf79a97149618bcaf0fefee1dfeb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:26.133271 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d8cdf79a97149618bcaf0fefee1dfeb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:26.133342 1980 raft_consensus.cc:2468] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:26.133371 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 7036a159e88b4171942dd238f8af3ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.133489 2118 raft_consensus.cc:3060] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.133649 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d8cdf79a97149618bcaf0fefee1dfeb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:26.133731 2376 tablet_replica.cc:333] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:26.133810 1981 raft_consensus.cc:3060] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.133872 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 6a3491b303f94fe2a2c838297f73d464 (DEFAULT_TABLE table=default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.133937 2376 raft_consensus.cc:2243] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.134166 2376 raft_consensus.cc:2272] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.134310 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a3491b303f94fe2a2c838297f73d464. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.134750 2376 ts_tablet_manager.cc:1916] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.136381 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet 3f60bcc5e4cb4fd19d2a5713f69814cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.137513 2118 raft_consensus.cc:2468] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:26.138044 2195 leader_election.cc:304] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:26.138737 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe (DEFAULT_TABLE table=default.Table_ffa4b05d8c84446297b336b55bbf5c6d [id=87349d9c6c764382b828e716e7854c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.138868 2440 raft_consensus.cc:2804] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:26.139021 2440 raft_consensus.cc:697] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:26.139123 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.139304 2440 consensus_queue.cc:237] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.141548 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.141759 2631 raft_consensus.cc:493] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.141814 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 461412f3ad1e4186b1302cdf5a624ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.141911 2631 raft_consensus.cc:515] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.142138 2096 tablet_service.cc:1505] Processing CreateTablet for tablet a6f9ae327aa4401baf66a737eaa93cf6 (DEFAULT_TABLE table=default.Table_828c16a96a47489f835a891f34d707b1 [id=432234d273ca4c9ea53c7a078c274637]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.142550 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f9ae327aa4401baf66a737eaa93cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.142542 2631 leader_election.cc:290] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:26.143646 2084 tablet_service.cc:1505] Processing CreateTablet for tablet ff242ef9541a42d793c4484190956da6 (DEFAULT_TABLE table=default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.144111 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff242ef9541a42d793c4484190956da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.149116 1981 raft_consensus.cc:2468] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:26.152572 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfed3523205548508d1101586ee3e2a9" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:26.157513 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 30c619ad37364747b6ebf97a73ef45a2 (DEFAULT_TABLE table=default.Table_ad4b4db01c7d4fe0a4e99f2f54cee0e8 [id=65a13a0482834ea6b52b71c953d61e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.158054 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30c619ad37364747b6ebf97a73ef45a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:26.161854 2194 leader_election.cc:343] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.173270 2377 ts_tablet_manager.cc:1929] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:26.173539 2377 log.cc:1199] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/0c647782049243d6bcd595f1a10d89c6
I20251026 02:56:26.174602 2377 ts_tablet_manager.cc:1950] T 0c647782049243d6bcd595f1a10d89c6 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:26.175217 2376 ts_tablet_manager.cc:1929] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:26.175407 2376 log.cc:1199] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/7036a159e88b4171942dd238f8af3ed3
I20251026 02:56:26.175930 2376 ts_tablet_manager.cc:1950] T 7036a159e88b4171942dd238f8af3ed3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:26.176400 2377 tablet_replica.cc:333] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:26.176599 2376 tablet_replica.cc:333] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:26.177264 2377 raft_consensus.cc:2243] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.177008 2376 raft_consensus.cc:2243] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.177650 2376 raft_consensus.cc:2272] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.181787 2376 ts_tablet_manager.cc:1916] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.182049 2372 ts_tablet_manager.cc:1929] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:26.182250 2372 log.cc:1199] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/ac19c5a847454a23ab16831198ff2a20
I20251026 02:56:26.182904 2372 ts_tablet_manager.cc:1950] T ac19c5a847454a23ab16831198ff2a20 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:26.159637 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f3ae3a377af46449c3faca339f86053 consensus state has no leader
I20251026 02:56:26.159900 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7801f073d5d94076af9299492f86fad2
W20251026 02:56:26.178293 2195 leader_election.cc:343] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: cfed3523205548508d1101586ee3e2a9
I20251026 02:56:26.186445 2377 raft_consensus.cc:2272] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.180716 2759 consensus_queue.cc:1048] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:26.161154 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet 1c1596dd4394419a8bb6ea6a84be9fea from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:26.186997 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 7801f073d5d94076af9299492f86fad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:26.187040 2377 ts_tablet_manager.cc:1916] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:26.129125 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3327d26a2c2c4d0b86d98151f5940bd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3327d26a2c2c4d0b86d98151f5940bd7 already in progress: creating tablet
W20251026 02:56:26.187625 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 58710.3us
I20251026 02:56:26.187685 2381 tablet_bootstrap.cc:492] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.187821 2381 ts_tablet_manager.cc:1403] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.095s user 0.006s sys 0.002s
I20251026 02:56:26.172415 1654 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_620d0c4122484e579155fe90d2d5c491 [id=6734e5731cc94e71b43c76512f649d57] alter complete (version 2)
I20251026 02:56:26.188634 2195 leader_election.cc:304] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.188784 2381 raft_consensus.cc:359] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.188992 2381 raft_consensus.cc:385] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.188990 2759 raft_consensus.cc:2749] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.189123 2381 raft_consensus.cc:740] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.189338 2381 consensus_queue.cc:260] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.190050 2381 ts_tablet_manager.cc:1434] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:26.190328 2381 tablet_bootstrap.cc:492] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.192346 2381 tablet_bootstrap.cc:654] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.194267 2768 tablet.cc:1722] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:26.205004 2646 raft_consensus.cc:493] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.205325 2646 raft_consensus.cc:515] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.205773 2375 tablet_bootstrap.cc:492] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.205938 2375 ts_tablet_manager.cc:1403] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.189s user 0.009s sys 0.002s
I20251026 02:56:26.206848 2381 tablet_bootstrap.cc:492] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.206775 2375 raft_consensus.cc:359] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.206941 2375 raft_consensus.cc:385] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.207031 2381 ts_tablet_manager.cc:1403] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.008s
I20251026 02:56:26.207042 2375 raft_consensus.cc:740] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.207592 2646 leader_election.cc:290] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.207913 2381 raft_consensus.cc:359] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.208036 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "740f7c7f186f45ceaa4c40e76419e3a1" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:26.208153 2381 raft_consensus.cc:385] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.208254 2381 raft_consensus.cc:740] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.208473 2381 consensus_queue.cc:260] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:26.208575 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d65b0d79e2e54af9810022b683c3288b consensus state has no leader
W20251026 02:56:26.208585 2053 leader_election.cc:343] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.209192 2381 ts_tablet_manager.cc:1434] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:26.209493 2381 tablet_bootstrap.cc:492] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.210543 1658 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 23fd1284c9cb41bd98209ccbfe342bca (table default.Table_620d0c4122484e579155fe90d2d5c491 [id=6734e5731cc94e71b43c76512f649d57]). Expected version 2 got 0
I20251026 02:56:26.209007 2375 consensus_queue.cc:260] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.211769 2381 tablet_bootstrap.cc:654] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.212291 2375 ts_tablet_manager.cc:1434] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251026 02:56:26.212982 2375 tablet_bootstrap.cc:492] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.213366 1660 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 23fd1284c9cb41bd98209ccbfe342bca (table default.Table_620d0c4122484e579155fe90d2d5c491 [id=6734e5731cc94e71b43c76512f649d57]). Expected version 2 got 0
I20251026 02:56:26.212663 2755 tablet.cc:1717] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539: Skipping requested alter to schema version 2, tablet already version 2
I20251026 02:56:26.215147 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "740f7c7f186f45ceaa4c40e76419e3a1" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:26.216231 2053 leader_election.cc:343] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.216400 2053 leader_election.cc:304] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.216969 2646 raft_consensus.cc:2749] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.217175 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ac2590e433d341b48b9303f46840d3e5" } new_table_name: "default.Table_7cc1906991304bd4980fe66c79599f73" modify_external_catalogs: true
I20251026 02:56:26.221733 2375 tablet_bootstrap.cc:654] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.223227 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a70d70abc3a34dac8cf9a747903fa78c
I20251026 02:56:26.224643 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet a70d70abc3a34dac8cf9a747903fa78c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:26.225858 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8667ffa25f2b478a987826d486400bbe
I20251026 02:56:26.226135 2377 ts_tablet_manager.cc:1929] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:26.226279 2377 log.cc:1199] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/20dec74440cf4bb9949c462ec51e9889
I20251026 02:56:26.226719 2377 ts_tablet_manager.cc:1950] T 20dec74440cf4bb9949c462ec51e9889 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:26.227138 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 8667ffa25f2b478a987826d486400bbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:26.227330 2375 tablet_bootstrap.cc:492] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.227500 2375 ts_tablet_manager.cc:1403] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
W20251026 02:56:26.227525 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d7ffdaeec2240dcbe4cafdb96073a7d already in progress: creating tablet
I20251026 02:56:26.227653 2768 tablet.cc:1717] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48: Skipping requested alter to schema version 2, tablet already version 2
I20251026 02:56:26.228046 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 7036a159e88b4171942dd238f8af3ed3 (table default.Table_7e5c0b5970974e12aba66398ec59430e [id=59edc20bb9be438c9ef00aafcd1c6376]) successfully deleted
I20251026 02:56:26.229409 2375 raft_consensus.cc:359] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.229650 2375 raft_consensus.cc:385] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.229727 2375 raft_consensus.cc:740] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.230573 1655 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet b24b92465bd2437a98e80b9020c66d46
I20251026 02:56:26.229223 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 6a3491b303f94fe2a2c838297f73d464 (DEFAULT_TABLE table=default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.233923 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a3491b303f94fe2a2c838297f73d464. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.234246 2375 consensus_queue.cc:260] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.235502 2381 tablet_bootstrap.cc:492] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.235864 2381 ts_tablet_manager.cc:1403] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.016s
I20251026 02:56:26.236541 2375 ts_tablet_manager.cc:1434] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20251026 02:56:26.236958 2381 raft_consensus.cc:359] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.237330 2381 raft_consensus.cc:385] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.237493 2381 raft_consensus.cc:740] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.237865 2381 consensus_queue.cc:260] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.238449 2375 tablet_bootstrap.cc:492] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.238745 2381 ts_tablet_manager.cc:1434] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:26.239084 2381 tablet_bootstrap.cc:492] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.240348 2375 tablet_bootstrap.cc:654] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.241156 2381 tablet_bootstrap.cc:654] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.242619 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 7801f073d5d94076af9299492f86fad2 (DEFAULT_TABLE table=default.Table_03a9f618938a45aab881c91a801cc32c [id=12b41b47b5d243188eb73bd83a83bbdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.243026 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7801f073d5d94076af9299492f86fad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.249702 2381 tablet_bootstrap.cc:492] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.249711 2375 tablet_bootstrap.cc:492] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.249861 2375 ts_tablet_manager.cc:1403] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20251026 02:56:26.249891 2381 ts_tablet_manager.cc:1403] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
W20251026 02:56:26.250006 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 155a23a93358487a9564bb5ac175b4eb consensus state has no leader
I20251026 02:56:26.250725 2381 raft_consensus.cc:359] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.250926 2381 raft_consensus.cc:385] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.251019 2381 raft_consensus.cc:740] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.251513 2761 tablet.cc:1717] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8: Skipping requested alter to schema version 2, tablet already version 2
I20251026 02:56:26.251609 2381 consensus_queue.cc:260] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.252115 2381 ts_tablet_manager.cc:1434] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:26.252734 2381 tablet_bootstrap.cc:492] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.253175 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet ac19c5a847454a23ab16831198ff2a20 (table default.Table_f8a501269b994d8c8864349d12974e16 [id=a48fa20d35e44980b60317a9c93b42d5]) successfully deleted
I20251026 02:56:26.253892 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 30c619ad37364747b6ebf97a73ef45a2 (DEFAULT_TABLE table=default.Table_ad4b4db01c7d4fe0a4e99f2f54cee0e8 [id=65a13a0482834ea6b52b71c953d61e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.254300 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30c619ad37364747b6ebf97a73ef45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.254536 2381 tablet_bootstrap.cc:654] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:26.255827 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d7ffdaeec2240dcbe4cafdb96073a7d already in progress: creating tablet
I20251026 02:56:26.256685 2230 tablet_service.cc:1505] Processing CreateTablet for tablet a6f9ae327aa4401baf66a737eaa93cf6 (DEFAULT_TABLE table=default.Table_828c16a96a47489f835a891f34d707b1 [id=432234d273ca4c9ea53c7a078c274637]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.257037 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f9ae327aa4401baf66a737eaa93cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.256973 2243 tablet_service.cc:1505] Processing CreateTablet for tablet ff242ef9541a42d793c4484190956da6 (DEFAULT_TABLE table=default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.257292 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff242ef9541a42d793c4484190956da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.250662 2375 raft_consensus.cc:359] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.258433 2375 raft_consensus.cc:385] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.258566 2375 raft_consensus.cc:740] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.258877 2375 consensus_queue.cc:260] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.261804 2375 ts_tablet_manager.cc:1434] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251026 02:56:26.264261 1962 tablet_service.cc:1505] Processing CreateTablet for tablet ff242ef9541a42d793c4484190956da6 (DEFAULT_TABLE table=default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.267446 2376 ts_tablet_manager.cc:1929] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:26.270040 2372 tablet_replica.cc:333] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:26.272080 2372 raft_consensus.cc:2243] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.274003 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 7801f073d5d94076af9299492f86fad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.274390 2372 raft_consensus.cc:2272] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.275087 2375 tablet_bootstrap.cc:492] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.280161 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c1596dd4394419a8bb6ea6a84be9fea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.280254 2375 tablet_bootstrap.cc:654] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.280818 2372 ts_tablet_manager.cc:1916] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.281684 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c647782049243d6bcd595f1a10d89c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 5caeb11322fe48348ebb523967152187) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.281967 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff242ef9541a42d793c4484190956da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.283592 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet a70d70abc3a34dac8cf9a747903fa78c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.283591 1961 tablet_service.cc:1505] Processing CreateTablet for tablet a6f9ae327aa4401baf66a737eaa93cf6 (DEFAULT_TABLE table=default.Table_828c16a96a47489f835a891f34d707b1 [id=432234d273ca4c9ea53c7a078c274637]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.283891 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet a70d70abc3a34dac8cf9a747903fa78c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.285687 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 8667ffa25f2b478a987826d486400bbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.286039 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 8667ffa25f2b478a987826d486400bbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.286118 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f9ae327aa4401baf66a737eaa93cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.287104 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 0c647782049243d6bcd595f1a10d89c6 (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
W20251026 02:56:26.287343 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 863ce9e1adb444b7b1399b968ea79bb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 863ce9e1adb444b7b1399b968ea79bb4 already in progress: creating tablet
W20251026 02:56:26.287559 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 9156314f98ac4ed9981f6f13fc585043 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9156314f98ac4ed9981f6f13fc585043 already in progress: creating tablet
I20251026 02:56:26.287673 2381 tablet_bootstrap.cc:492] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
W20251026 02:56:26.287760 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 634d913c98c842a49d97a54f23752d26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 634d913c98c842a49d97a54f23752d26 already in progress: creating tablet
I20251026 02:56:26.287837 2381 ts_tablet_manager.cc:1403] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.005s
W20251026 02:56:26.287935 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3327d26a2c2c4d0b86d98151f5940bd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3327d26a2c2c4d0b86d98151f5940bd7 already in progress: creating tablet
I20251026 02:56:26.288794 2381 raft_consensus.cc:359] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.289073 2381 raft_consensus.cc:385] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.289160 2381 raft_consensus.cc:740] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.289244 2375 tablet_bootstrap.cc:492] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.289400 2375 ts_tablet_manager.cc:1403] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20251026 02:56:26.289376 2381 consensus_queue.cc:260] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.289855 2381 ts_tablet_manager.cc:1434] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:26.290648 2381 tablet_bootstrap.cc:492] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.292485 2381 tablet_bootstrap.cc:654] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.290133 2375 raft_consensus.cc:359] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.292997 2375 raft_consensus.cc:385] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.293094 2375 raft_consensus.cc:740] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.293370 2375 consensus_queue.cc:260] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.295817 2375 ts_tablet_manager.cc:1434] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251026 02:56:26.297227 2376 log.cc:1199] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3f60bcc5e4cb4fd19d2a5713f69814cd
I20251026 02:56:26.297381 2375 tablet_bootstrap.cc:492] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.297734 2376 ts_tablet_manager.cc:1950] T 3f60bcc5e4cb4fd19d2a5713f69814cd P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:26.298907 2376 tablet_replica.cc:333] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:26.299173 2376 raft_consensus.cc:2243] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.299443 2376 raft_consensus.cc:2272] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.300051 2376 ts_tablet_manager.cc:1916] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.303972 2381 tablet_bootstrap.cc:492] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.304178 2381 ts_tablet_manager.cc:1403] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20251026 02:56:26.304167 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 7801f073d5d94076af9299492f86fad2 (DEFAULT_TABLE table=default.Table_03a9f618938a45aab881c91a801cc32c [id=12b41b47b5d243188eb73bd83a83bbdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.304715 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7801f073d5d94076af9299492f86fad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.304997 2381 raft_consensus.cc:359] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.305207 2381 raft_consensus.cc:385] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.305276 2381 raft_consensus.cc:740] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.305531 2381 consensus_queue.cc:260] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.306034 2381 ts_tablet_manager.cc:1434] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:26.310150 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_14cbc77849c244ae98b9a3afc5a8fdc0" } new_table_name: "default.Table_f7c216f3ebd34c718ceef042a435b4cd" modify_external_catalogs: true
I20251026 02:56:26.311252 2381 tablet_bootstrap.cc:492] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.313405 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet 7801f073d5d94076af9299492f86fad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.313550 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 7439c051ca634fbe8b511d47ccede2ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.313633 2381 tablet_bootstrap.cc:654] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.313661 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 30c619ad37364747b6ebf97a73ef45a2 (DEFAULT_TABLE table=default.Table_ad4b4db01c7d4fe0a4e99f2f54cee0e8 [id=65a13a0482834ea6b52b71c953d61e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.314141 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30c619ad37364747b6ebf97a73ef45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.314383 2375 tablet_bootstrap.cc:654] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:26.321625 2194 consensus_peers.cc:597] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:26.322263 2378 tablet_bootstrap.cc:492] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:26.322541 2378 ts_tablet_manager.cc:1403] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.291s user 0.010s sys 0.002s
I20251026 02:56:26.323251 2375 tablet_bootstrap.cc:492] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.323451 2375 ts_tablet_manager.cc:1403] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.002s
I20251026 02:56:26.323722 2378 raft_consensus.cc:359] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.323989 2378 raft_consensus.cc:385] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251026 02:56:26.324059 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22e4a433801a4cb58062d3e76c760b69 consensus state has no leader
I20251026 02:56:26.324138 2378 raft_consensus.cc:740] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.328081 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ad7f0263e7ad4bdea1b9805522ec23db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.328961 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad7f0263e7ad4bdea1b9805522ec23db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.329638 2381 tablet_bootstrap.cc:492] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.329797 2381 ts_tablet_manager.cc:1403] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20251026 02:56:26.329934 2440 consensus_queue.cc:1048] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:26.330535 2381 raft_consensus.cc:359] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.330689 2381 raft_consensus.cc:385] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.330756 2381 raft_consensus.cc:740] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.331275 2381 consensus_queue.cc:260] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.331916 2381 ts_tablet_manager.cc:1434] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:26.327698 2375 raft_consensus.cc:359] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.332171 2375 raft_consensus.cc:385] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.332237 2381 tablet_bootstrap.cc:492] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.332314 2375 raft_consensus.cc:740] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.332540 2375 consensus_queue.cc:260] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.334033 2381 tablet_bootstrap.cc:654] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.334069 2375 ts_tablet_manager.cc:1434] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20251026 02:56:26.335206 2375 tablet_bootstrap.cc:492] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.338114 2375 tablet_bootstrap.cc:654] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.347743 2381 tablet_bootstrap.cc:492] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.347924 2381 ts_tablet_manager.cc:1403] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.006s
I20251026 02:56:26.348713 2381 raft_consensus.cc:359] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.348919 2381 raft_consensus.cc:385] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.349004 2381 raft_consensus.cc:740] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.349226 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 155a23a93358487a9564bb5ac175b4eb from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:26.349247 2381 consensus_queue.cc:260] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.349761 2381 ts_tablet_manager.cc:1434] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:26.349864 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet a2bfbe624075463f8b90ed4ec66b784c from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:26.350131 2381 tablet_bootstrap.cc:492] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.351328 2375 tablet_bootstrap.cc:492] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.351500 2375 ts_tablet_manager.cc:1403] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20251026 02:56:26.352382 2759 raft_consensus.cc:493] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.352386 2375 raft_consensus.cc:359] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.352551 2759 raft_consensus.cc:515] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.352619 2375 raft_consensus.cc:385] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.352694 2375 raft_consensus.cc:740] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.352978 2375 consensus_queue.cc:260] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.353140 2381 tablet_bootstrap.cc:654] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.353267 2759 leader_election.cc:290] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:26.354293 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1b6257a3c8c4e1a8144ea45ff2a2a96" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:26.354631 2195 leader_election.cc:343] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: c1b6257a3c8c4e1a8144ea45ff2a2a96
I20251026 02:56:26.356451 2375 ts_tablet_manager.cc:1434] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251026 02:56:26.361325 2378 consensus_queue.cc:260] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.362048 2378 ts_tablet_manager.cc:1434] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.039s user 0.000s sys 0.002s
I20251026 02:56:26.362921 2378 tablet_bootstrap.cc:492] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:26.363859 2375 tablet_bootstrap.cc:492] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.365365 2378 tablet_bootstrap.cc:654] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.367233 2375 tablet_bootstrap.cc:654] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.372417 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a2bfbe624075463f8b90ed4ec66b784c
I20251026 02:56:26.372961 1663 catalog_manager.cc:6719] ReplaceTablet: tablet a2bfbe624075463f8b90ed4ec66b784c deleted and replaced by tablet ec13c5386abe414ba25589e03d5b850a
I20251026 02:56:26.373672 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet a2bfbe624075463f8b90ed4ec66b784c with delete_type TABLET_DATA_DELETED (replaced by tablet ec13c5386abe414ba25589e03d5b850a) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:26.376691 2377 tablet_replica.cc:333] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:26.376947 2377 raft_consensus.cc:2243] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.376958 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 8dbc12d49b094dbeadad2e76916ba9af from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:26.377125 2377 raft_consensus.cc:2272] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.377737 2377 ts_tablet_manager.cc:1916] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.377979 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_61b2d44fec564f0a8b01015ef7ee44ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.378593 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61b2d44fec564f0a8b01015ef7ee44ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.379493 2381 tablet_bootstrap.cc:492] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.379681 2381 ts_tablet_manager.cc:1403] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.012s
W20251026 02:56:26.380451 2194 leader_election.cc:343] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.380592 2440 raft_consensus.cc:493] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.380749 2440 raft_consensus.cc:515] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.381436 2440 leader_election.cc:290] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:26.380623 2194 leader_election.cc:304] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.381891 2440 raft_consensus.cc:2749] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.381790 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78a624d839934d3089b64e397d4360af" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:26.382136 2118 raft_consensus.cc:2468] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:26.382530 2195 leader_election.cc:304] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:26.381184 2381 raft_consensus.cc:359] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.382774 2381 raft_consensus.cc:385] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.382854 2381 raft_consensus.cc:740] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.382813 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78a624d839934d3089b64e397d4360af" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:26.383111 2381 consensus_queue.cc:260] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.383374 2759 raft_consensus.cc:2804] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:26.383514 2759 raft_consensus.cc:493] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:26.383615 2759 raft_consensus.cc:3060] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.383621 2381 ts_tablet_manager.cc:1434] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251026 02:56:26.383925 2381 tablet_bootstrap.cc:492] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:26.384156 2194 leader_election.cc:343] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.385784 2381 tablet_bootstrap.cc:654] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.386674 2759 raft_consensus.cc:515] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.387852 2759 leader_election.cc:290] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:26.388190 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78a624d839934d3089b64e397d4360af" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:26.388388 2118 raft_consensus.cc:3060] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.393183 2381 tablet_bootstrap.cc:492] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.393443 2381 ts_tablet_manager.cc:1403] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20251026 02:56:26.393445 2376 ts_tablet_manager.cc:1929] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:26.393579 2376 log.cc:1199] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/1c1596dd4394419a8bb6ea6a84be9fea
I20251026 02:56:26.393950 2376 ts_tablet_manager.cc:1950] T 1c1596dd4394419a8bb6ea6a84be9fea P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:26.394789 2381 raft_consensus.cc:359] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.395139 2381 raft_consensus.cc:385] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.395296 2381 raft_consensus.cc:740] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.395696 2381 consensus_queue.cc:260] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.395911 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78a624d839934d3089b64e397d4360af" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:26.396735 2375 tablet_bootstrap.cc:492] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.396900 2375 ts_tablet_manager.cc:1403] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
W20251026 02:56:26.397393 2194 leader_election.cc:343] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.397627 2375 raft_consensus.cc:359] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.397866 2375 raft_consensus.cc:385] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.397995 2375 raft_consensus.cc:740] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.398228 2375 consensus_queue.cc:260] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.398833 2375 ts_tablet_manager.cc:1434] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:26.399168 2375 tablet_bootstrap.cc:492] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.401082 2375 tablet_bootstrap.cc:654] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.401373 2376 tablet_replica.cc:333] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:26.401661 2376 raft_consensus.cc:2243] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.401903 2376 raft_consensus.cc:2272] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.402052 2118 raft_consensus.cc:2468] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:26.402582 2195 leader_election.cc:304] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.403096 2759 raft_consensus.cc:2804] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:26.403275 2759 raft_consensus.cc:697] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:26.403676 2759 consensus_queue.cc:237] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.403673 2381 ts_tablet_manager.cc:1434] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20251026 02:56:26.404698 2376 ts_tablet_manager.cc:1916] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.407037 2381 tablet_bootstrap.cc:492] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.408090 2440 raft_consensus.cc:493] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.408349 2440 raft_consensus.cc:515] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.409053 2440 leader_election.cc:290] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:26.410480 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "461412f3ad1e4186b1302cdf5a624ed4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:26.411178 2194 leader_election.cc:343] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:26.412081 2195 leader_election.cc:343] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 461412f3ad1e4186b1302cdf5a624ed4
I20251026 02:56:26.413463 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a5ff6167974b47798ffe2b3cd583b50b" } new_table_name: "default.Table_12a3f7cac0614b76888ec0101475ab29" modify_external_catalogs: true
W20251026 02:56:26.420097 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a08e8b7eb1949408f2151f2d7257e9d consensus state has no leader
I20251026 02:56:26.422974 2195 leader_election.cc:304] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.423686 2643 raft_consensus.cc:2749] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.423903 2381 tablet_bootstrap.cc:654] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.423024 2440 raft_consensus.cc:493] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.424345 2440 raft_consensus.cc:515] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.424983 2440 leader_election.cc:290] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
W20251026 02:56:26.426008 2195 leader_election.cc:343] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ea910e98d17340efb816a8aa90f0ba7e
I20251026 02:56:26.427489 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea910e98d17340efb816a8aa90f0ba7e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:26.427704 1981 raft_consensus.cc:2468] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:26.429491 2440 consensus_queue.cc:1048] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:26.431138 2194 leader_election.cc:304] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:26.431512 2440 raft_consensus.cc:2804] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:26.431599 2440 raft_consensus.cc:493] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:26.431696 2440 raft_consensus.cc:3060] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.434444 2440 raft_consensus.cc:515] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:26.435788 2195 leader_election.cc:343] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ea910e98d17340efb816a8aa90f0ba7e
I20251026 02:56:26.436522 2440 leader_election.cc:290] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:26.437177 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea910e98d17340efb816a8aa90f0ba7e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:26.437376 1981 raft_consensus.cc:3060] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:26.438556 1648 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 155a23a93358487a9564bb5ac175b4eb (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 155a23a93358487a9564bb5ac175b4eb consensus state has no leader
I20251026 02:56:26.438941 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet ea910e98d17340efb816a8aa90f0ba7e from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:26.439687 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_81e9f54ed3934b6b80f7c10d0be54d9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.440454 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81e9f54ed3934b6b80f7c10d0be54d9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.440584 1981 raft_consensus.cc:2468] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:26.441093 2194 leader_election.cc:304] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:26.441939 2440 raft_consensus.cc:2804] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:26.443065 2440 raft_consensus.cc:697] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:26.443564 2440 consensus_queue.cc:237] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.443849 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_18e8723148dd46e6a56e5716a15d01bc" } modify_external_catalogs: true
I20251026 02:56:26.444068 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_18e8723148dd46e6a56e5716a15d01bc" } modify_external_catalogs: true
I20251026 02:56:26.444945 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_15e626aa074c4736a61a03be25b64350"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.445611 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15e626aa074c4736a61a03be25b64350 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.448992 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b555f404492a47808d8a9a168a45ef2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:26.420195 2377 ts_tablet_manager.cc:1929] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251026 02:56:26.449609 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b555f404492a47808d8a9a168a45ef2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.449741 2377 log.cc:1199] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7176cbe8f64546b7871d047c0e0e5b2a
I20251026 02:56:26.450197 2377 ts_tablet_manager.cc:1950] T 7176cbe8f64546b7871d047c0e0e5b2a P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:26.452369 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1bc1610289a448d99b780c0e289b7d58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.452832 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bc1610289a448d99b780c0e289b7d58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.455742 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5ad87a5fcdd745529b77d2ffd8c2a4ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.456574 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ad87a5fcdd745529b77d2ffd8c2a4ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.459208 2376 ts_tablet_manager.cc:1929] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:26.459614 2377 tablet_replica.cc:333] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:26.460171 2375 tablet_bootstrap.cc:492] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.460386 2375 ts_tablet_manager.cc:1403] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.061s user 0.013s sys 0.012s
I20251026 02:56:26.461426 2375 raft_consensus.cc:359] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.461621 2375 raft_consensus.cc:385] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.461691 2375 raft_consensus.cc:740] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.461917 2375 consensus_queue.cc:260] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.462637 2375 ts_tablet_manager.cc:1434] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:26.462956 2375 tablet_bootstrap.cc:492] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.465114 2375 tablet_bootstrap.cc:654] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.466792 1654 catalog_manager.cc:5649] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:26.467373 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 1d8cdf79a97149618bcaf0fefee1dfeb (table default.Table_424693529d0d4c829755adc2c9835c0e [id=690ec15818d34cdfa990c8fcd36a19cd]). Expected version 1 got 0
I20251026 02:56:26.469575 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 340bf8c78a88413299b61a52f0a578f3 (table default.Table_1361fbf89aa04321a7876c1e43e27089 [id=c162ef89d2f14e91a24a6d8c2c37c261]). Expected version 4 got 0
I20251026 02:56:26.466706 2378 tablet_bootstrap.cc:492] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:26.473095 2646 raft_consensus.cc:493] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.473309 2646 raft_consensus.cc:515] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.473896 2378 ts_tablet_manager.cc:1403] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.108s user 0.009s sys 0.024s
I20251026 02:56:26.474042 2646 leader_election.cc:290] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.474983 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a821ed8a686c452a975acb96b5c74778" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:26.475682 2375 tablet_bootstrap.cc:492] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.475803 2375 ts_tablet_manager.cc:1403] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
W20251026 02:56:26.476720 2053 leader_election.cc:343] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.478088 2375 raft_consensus.cc:359] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.478256 2375 raft_consensus.cc:385] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.478325 2375 raft_consensus.cc:740] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.478515 2375 consensus_queue.cc:260] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.480391 2375 ts_tablet_manager.cc:1434] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251026 02:56:26.480804 2378 raft_consensus.cc:359] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.481669 2375 tablet_bootstrap.cc:492] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.481683 2759 raft_consensus.cc:493] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.481881 2378 raft_consensus.cc:385] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.481861 2759 raft_consensus.cc:515] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.481987 2378 raft_consensus.cc:740] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.482558 2759 leader_election.cc:290] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:26.483279 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8667ffa25f2b478a987826d486400bbe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:26.483502 2118 raft_consensus.cc:2468] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:26.483561 2378 consensus_queue.cc:260] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.483834 2646 raft_consensus.cc:493] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.483963 2646 raft_consensus.cc:515] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.484153 2378 ts_tablet_manager.cc:1434] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251026 02:56:26.484566 2646 leader_election.cc:290] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.485558 2195 leader_election.cc:304] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:26.485919 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "903e65fa8c214dc9a50cb97cab101d22" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:26.486128 2262 raft_consensus.cc:2468] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:26.486392 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_dc751f63612b4379a8d8069bc750d1f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:26.486698 2375 tablet_bootstrap.cc:654] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.486743 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "903e65fa8c214dc9a50cb97cab101d22" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:26.487007 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc751f63612b4379a8d8069bc750d1f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.487066 2053 leader_election.cc:304] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:26.487298 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a821ed8a686c452a975acb96b5c74778" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:26.487387 2053 leader_election.cc:343] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:26.487709 2053 leader_election.cc:343] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.487826 2053 leader_election.cc:304] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.488396 2646 raft_consensus.cc:2804] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:26.488528 2646 raft_consensus.cc:493] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:26.488632 2646 raft_consensus.cc:3060] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.491605 2646 raft_consensus.cc:515] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.492359 2758 raft_consensus.cc:2749] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.492444 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet a2bfbe624075463f8b90ed4ec66b784c with delete_type TABLET_DATA_DELETED (replaced by tablet ec13c5386abe414ba25589e03d5b850a) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.493106 2643 raft_consensus.cc:2804] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:26.493186 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet a2bfbe624075463f8b90ed4ec66b784c with delete_type TABLET_DATA_DELETED (replaced by tablet ec13c5386abe414ba25589e03d5b850a) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.493230 2643 raft_consensus.cc:493] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:26.493340 2643 raft_consensus.cc:3060] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.495158 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8667ffa25f2b478a987826d486400bbe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:26.495342 2378 tablet_bootstrap.cc:492] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:26.495260 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 (DEFAULT_TABLE table=default.Table_2dfe8bcc7f1e4933a4174bdab19cffd5 [id=65d576adb71949cc97666624935adbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.495674 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:26.495679 2194 leader_election.cc:343] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.496070 2643 raft_consensus.cc:515] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.497000 2378 tablet_bootstrap.cc:654] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.497212 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "903e65fa8c214dc9a50cb97cab101d22" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:26.497400 2262 raft_consensus.cc:3060] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.497354 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "903e65fa8c214dc9a50cb97cab101d22" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:26.497884 2053 leader_election.cc:343] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.498780 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 8862cd48a5744c99b32f365cb77fa7c4 (DEFAULT_TABLE table=default.Table_f12577c6a60b4858b2e778692827a023 [id=604570227c014f66aeae89d93edaaafe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.499147 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8862cd48a5744c99b32f365cb77fa7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.499310 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3bc5fdbe07c348cf8f30654554a8b95a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.499861 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bc5fdbe07c348cf8f30654554a8b95a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.499998 2372 ts_tablet_manager.cc:1929] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:26.500162 2372 log.cc:1199] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7dec83bfb9e946db90c10d526af2dfca
I20251026 02:56:26.500612 2372 ts_tablet_manager.cc:1950] T 7dec83bfb9e946db90c10d526af2dfca P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:26.502907 2643 leader_election.cc:290] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:26.503626 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8667ffa25f2b478a987826d486400bbe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:26.504523 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8667ffa25f2b478a987826d486400bbe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:26.504760 2118 raft_consensus.cc:3060] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.507611 2118 raft_consensus.cc:2468] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:26.508425 2194 leader_election.cc:343] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.508702 2195 leader_election.cc:304] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.509186 2643 raft_consensus.cc:2804] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:26.509333 2643 raft_consensus.cc:697] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:26.509614 2643 consensus_queue.cc:237] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.510766 2375 tablet_bootstrap.cc:492] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.510921 2375 ts_tablet_manager.cc:1403] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.001s
I20251026 02:56:26.492204 2646 leader_election.cc:290] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.511731 2262 raft_consensus.cc:2468] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:26.512302 2053 leader_election.cc:304] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.512634 2646 raft_consensus.cc:2804] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:26.512866 2646 raft_consensus.cc:697] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:26.513146 2646 consensus_queue.cc:237] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.513999 2376 log.cc:1199] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/0c647782049243d6bcd595f1a10d89c6
I20251026 02:56:26.514593 2376 ts_tablet_manager.cc:1950] T 0c647782049243d6bcd595f1a10d89c6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:26.517880 2376 tablet_replica.cc:333] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:26.518357 2376 raft_consensus.cc:2243] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.518874 2376 raft_consensus.cc:2272] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.519649 2376 ts_tablet_manager.cc:1916] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.525022 2378 tablet_bootstrap.cc:492] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:26.525173 2378 ts_tablet_manager.cc:1403] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20251026 02:56:26.525933 2378 raft_consensus.cc:359] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.526137 2378 raft_consensus.cc:385] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.526229 2378 raft_consensus.cc:740] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.526448 2378 consensus_queue.cc:260] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.526991 2378 ts_tablet_manager.cc:1434] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:26.528101 2378 tablet_bootstrap.cc:492] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:26.530499 2378 tablet_bootstrap.cc:654] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.531765 2376 ts_tablet_manager.cc:1929] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:26.531973 2376 log.cc:1199] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/7439c051ca634fbe8b511d47ccede2ff
I20251026 02:56:26.532588 2376 ts_tablet_manager.cc:1950] T 7439c051ca634fbe8b511d47ccede2ff P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:26.534910 2376 tablet_replica.cc:333] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:26.535363 2376 raft_consensus.cc:2243] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.535830 2376 raft_consensus.cc:2272] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.537086 2376 ts_tablet_manager.cc:1916] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.539816 2381 tablet_bootstrap.cc:492] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.540004 2381 ts_tablet_manager.cc:1403] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.135s user 0.006s sys 0.026s
I20251026 02:56:26.541081 2381 raft_consensus.cc:359] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.541393 2381 raft_consensus.cc:385] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.541512 2381 raft_consensus.cc:740] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.541738 2381 consensus_queue.cc:260] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.542429 2381 ts_tablet_manager.cc:1434] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:26.542871 2381 tablet_bootstrap.cc:492] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.545572 2381 tablet_bootstrap.cc:654] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.550218 2376 ts_tablet_manager.cc:1929] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:26.553217 2376 log.cc:1199] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/a2bfbe624075463f8b90ed4ec66b784c
I20251026 02:56:26.553920 2376 ts_tablet_manager.cc:1950] T a2bfbe624075463f8b90ed4ec66b784c P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:26.555673 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b4f29de6c177447f81779bfa9e30ac2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.556378 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4f29de6c177447f81779bfa9e30ac2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.557767 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fc5e859548014044852b49ab2b64bd13" } new_table_name: "default.Table_672ee2e3775346679182f116bb01cece" modify_external_catalogs: true
I20251026 02:56:26.558719 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b78fbd41028a4314ae6a81090f004797" } new_table_name: "default.Table_f83c29969baa4d62bca3154cc3d2cc33" modify_external_catalogs: true
I20251026 02:56:26.559177 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e9da6904da2c4d15bb17df7e9bbb3221" } modify_external_catalogs: true
I20251026 02:56:26.560444 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e9da6904da2c4d15bb17df7e9bbb3221" } modify_external_catalogs: true
I20251026 02:56:26.560949 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d6690e6bf67454398637d9908b9c652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.561328 2376 tablet_replica.cc:333] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:26.561563 2376 raft_consensus.cc:2243] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.561828 2376 raft_consensus.cc:2272] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.562460 2376 ts_tablet_manager.cc:1916] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.563231 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 5c6e3e47b3e7421e945cfbe671b06abe (DEFAULT_TABLE table=default.Table_4f8511a57eb54e5eaf42a6361676ff21 [id=a305da1e3a694b32aff05dbd6c58a6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.564101 2084 tablet_service.cc:1505] Processing CreateTablet for tablet e701d4023fee45d18c48997aab3227c2 (DEFAULT_TABLE table=default.Table_b78fbd41028a4314ae6a81090f004797 [id=de836edc225f41d8b0c68748936306ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.565666 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 882cd1680aa140f6a63a961cd4aec25a (DEFAULT_TABLE table=default.Table_6281fc8054c54d01b0376df4725d1adc [id=cc7a8c37a3b342d988872e3baa16a26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.567203 2381 tablet_bootstrap.cc:492] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.567236 2377 raft_consensus.cc:2243] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:26.568888 2375 raft_consensus.cc:359] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.569126 2375 raft_consensus.cc:385] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.569209 2375 raft_consensus.cc:740] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.569480 2375 consensus_queue.cc:260] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.569674 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c6e3e47b3e7421e945cfbe671b06abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.570040 2375 ts_tablet_manager.cc:1434] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.059s user 0.001s sys 0.001s
I20251026 02:56:26.572839 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_bcac335a2c9a479fb5dea86b2aeffde6" } modify_external_catalogs: true
I20251026 02:56:26.573026 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3f60bcc5e4cb4fd19d2a5713f69814cd (table default.Table_d029698b06db4a6f807451acb73a3680 [id=2ae19615aa104e2b8791b86755e1184f]) successfully deleted
I20251026 02:56:26.573071 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_bcac335a2c9a479fb5dea86b2aeffde6" } modify_external_catalogs: true
I20251026 02:56:26.573493 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 1c1596dd4394419a8bb6ea6a84be9fea (table default.Table_4a7c7fde2faf424590cdf3e2ddf0c1a5 [id=d9c0bfe3ef1145189ccef147212f787b]) successfully deleted
I20251026 02:56:26.573678 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 0c647782049243d6bcd595f1a10d89c6 (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
W20251026 02:56:26.573858 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet a70d70abc3a34dac8cf9a747903fa78c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a70d70abc3a34dac8cf9a747903fa78c already in progress: creating tablet
W20251026 02:56:26.574096 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 8667ffa25f2b478a987826d486400bbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8667ffa25f2b478a987826d486400bbe already in progress: creating tablet
W20251026 02:56:26.574270 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7801f073d5d94076af9299492f86fad2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7801f073d5d94076af9299492f86fad2 already in progress: creating tablet
I20251026 02:56:26.574434 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 7439c051ca634fbe8b511d47ccede2ff (table default.Table_be7d5ddd5e4e4b15862de09dc573f7f9 [id=354a2ac9115e4f2385537f1c16992882]) successfully deleted
I20251026 02:56:26.574750 2375 tablet_bootstrap.cc:492] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.575830 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882cd1680aa140f6a63a961cd4aec25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.577457 2375 tablet_bootstrap.cc:654] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.578063 2376 ts_tablet_manager.cc:1929] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:26.578470 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]) successfully deleted
I20251026 02:56:26.578883 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 20dec74440cf4bb9949c462ec51e9889 (table default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]) successfully deleted
I20251026 02:56:26.579200 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7176cbe8f64546b7871d047c0e0e5b2a (table default.Table_c46d7af7b60a48188efbea88a1b7a72b [id=93bef2a8d7b649e880042bfdbb1e08ac]) successfully deleted
I20251026 02:56:26.579500 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7dec83bfb9e946db90c10d526af2dfca (table default.Table_90929641047f40f1973c0e399816b5ef [id=327c67aeb10146e08119f7f3a7a11769]) successfully deleted
I20251026 02:56:26.579581 2376 log.cc:1199] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3d6690e6bf67454398637d9908b9c652
I20251026 02:56:26.579450 2378 tablet_bootstrap.cc:492] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:26.580173 2378 ts_tablet_manager.cc:1403] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.012s
I20251026 02:56:26.582033 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet ea910e98d17340efb816a8aa90f0ba7e with delete_type TABLET_DATA_DELETED (replaced by tablet 1370b9ad917344c093e5da3d0e9b1cf0) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.582474 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 (DEFAULT_TABLE table=default.Table_2dfe8bcc7f1e4933a4174bdab19cffd5 [id=65d576adb71949cc97666624935adbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.582840 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.582804 2096 tablet_service.cc:1505] Processing CreateTablet for tablet ed836bb7519e4b0290e6547fa57128bc (DEFAULT_TABLE table=default.Table_ec68beb934274468a4f028b3dbaf2cc0 [id=658f4d8e944a4221a722376dc2137896]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.583658 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 7177931353164b2aa82c00b2807ccd63 (DEFAULT_TABLE table=default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 [id=7de20c0869584b56b936144d62202444]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.584519 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 8862cd48a5744c99b32f365cb77fa7c4 (DEFAULT_TABLE table=default.Table_f12577c6a60b4858b2e778692827a023 [id=604570227c014f66aeae89d93edaaafe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.584877 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8862cd48a5744c99b32f365cb77fa7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.585672 2377 pending_rounds.cc:70] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:26.585747 2377 pending_rounds.cc:77] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888494878023680 op_type: NO_OP noop_request { }
I20251026 02:56:26.585882 2377 raft_consensus.cc:2889] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:26.585978 2377 raft_consensus.cc:2272] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.588869 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 882fe81da21344ed99ea735fa79f73c0 (DEFAULT_TABLE table=default.Table_18e8723148dd46e6a56e5716a15d01bc [id=156e8ddc92b1471e8dce064bcb23306b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.589885 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 5c6e3e47b3e7421e945cfbe671b06abe (DEFAULT_TABLE table=default.Table_4f8511a57eb54e5eaf42a6361676ff21 [id=a305da1e3a694b32aff05dbd6c58a6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.590312 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c6e3e47b3e7421e945cfbe671b06abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.590667 2381 ts_tablet_manager.cc:1403] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.004s
I20251026 02:56:26.590709 2100 tablet_service.cc:1505] Processing CreateTablet for tablet b1ca5b8379c94deab1130bfe8bdc87b0 (DEFAULT_TABLE table=default.Table_85c523c971b34e68bed4afa5d1aac450 [id=32c06e4782a8441c8d1614e81277b52e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.591351 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 8862cd48a5744c99b32f365cb77fa7c4 (DEFAULT_TABLE table=default.Table_f12577c6a60b4858b2e778692827a023 [id=604570227c014f66aeae89d93edaaafe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.592236 2381 raft_consensus.cc:359] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.592494 1953 tablet_service.cc:1505] Processing CreateTablet for tablet e701d4023fee45d18c48997aab3227c2 (DEFAULT_TABLE table=default.Table_b78fbd41028a4314ae6a81090f004797 [id=de836edc225f41d8b0c68748936306ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.592723 2381 raft_consensus.cc:385] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.592813 2381 raft_consensus.cc:740] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.592868 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e701d4023fee45d18c48997aab3227c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.593470 2375 tablet_bootstrap.cc:492] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.593595 2375 ts_tablet_manager.cc:1403] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20251026 02:56:26.594028 2381 consensus_queue.cc:260] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.594555 2381 ts_tablet_manager.cc:1434] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251026 02:56:26.594517 2375 raft_consensus.cc:359] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.594638 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 5c6e3e47b3e7421e945cfbe671b06abe (DEFAULT_TABLE table=default.Table_4f8511a57eb54e5eaf42a6361676ff21 [id=a305da1e3a694b32aff05dbd6c58a6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.594699 2375 raft_consensus.cc:385] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.594784 2375 raft_consensus.cc:740] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.593233 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 (DEFAULT_TABLE table=default.Table_2dfe8bcc7f1e4933a4174bdab19cffd5 [id=65d576adb71949cc97666624935adbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.595489 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 882cd1680aa140f6a63a961cd4aec25a (DEFAULT_TABLE table=default.Table_6281fc8054c54d01b0376df4725d1adc [id=cc7a8c37a3b342d988872e3baa16a26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.595891 2231 tablet_service.cc:1505] Processing CreateTablet for tablet ed836bb7519e4b0290e6547fa57128bc (DEFAULT_TABLE table=default.Table_ec68beb934274468a4f028b3dbaf2cc0 [id=658f4d8e944a4221a722376dc2137896]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.596612 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 882cd1680aa140f6a63a961cd4aec25a (DEFAULT_TABLE table=default.Table_6281fc8054c54d01b0376df4725d1adc [id=cc7a8c37a3b342d988872e3baa16a26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.597064 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882cd1680aa140f6a63a961cd4aec25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.596969 2375 consensus_queue.cc:260] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.597055 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 882fe81da21344ed99ea735fa79f73c0 (DEFAULT_TABLE table=default.Table_18e8723148dd46e6a56e5716a15d01bc [id=156e8ddc92b1471e8dce064bcb23306b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.597393 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8862cd48a5744c99b32f365cb77fa7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.597697 2375 ts_tablet_manager.cc:1434] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:26.598996 2236 tablet_service.cc:1505] Processing CreateTablet for tablet e701d4023fee45d18c48997aab3227c2 (DEFAULT_TABLE table=default.Table_b78fbd41028a4314ae6a81090f004797 [id=de836edc225f41d8b0c68748936306ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.599963 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 7177931353164b2aa82c00b2807ccd63 (DEFAULT_TABLE table=default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 [id=7de20c0869584b56b936144d62202444]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.600384 2232 tablet_service.cc:1505] Processing CreateTablet for tablet b1ca5b8379c94deab1130bfe8bdc87b0 (DEFAULT_TABLE table=default.Table_85c523c971b34e68bed4afa5d1aac450 [id=32c06e4782a8441c8d1614e81277b52e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.600642 2238 tablet_service.cc:1505] Processing CreateTablet for tablet de8799a5af2b42cc92ccc0195cff98e7 (DEFAULT_TABLE table=default.Table_fc5e859548014044852b49ab2b64bd13 [id=7885bd43f8a245b0a3010a63dbd535f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.600628 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 7177931353164b2aa82c00b2807ccd63 (DEFAULT_TABLE table=default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 [id=7de20c0869584b56b936144d62202444]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.601126 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7177931353164b2aa82c00b2807ccd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.602723 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e701d4023fee45d18c48997aab3227c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.603381 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882fe81da21344ed99ea735fa79f73c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.604624 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7177931353164b2aa82c00b2807ccd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.604669 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 882fe81da21344ed99ea735fa79f73c0 (DEFAULT_TABLE table=default.Table_18e8723148dd46e6a56e5716a15d01bc [id=156e8ddc92b1471e8dce064bcb23306b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.605124 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882fe81da21344ed99ea735fa79f73c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.605217 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1ca5b8379c94deab1130bfe8bdc87b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.605715 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8799a5af2b42cc92ccc0195cff98e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.606516 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.607357 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c6e3e47b3e7421e945cfbe671b06abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.608045 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed836bb7519e4b0290e6547fa57128bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.609050 2381 tablet_bootstrap.cc:492] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.608995 2086 tablet_service.cc:1505] Processing CreateTablet for tablet de8799a5af2b42cc92ccc0195cff98e7 (DEFAULT_TABLE table=default.Table_fc5e859548014044852b49ab2b64bd13 [id=7885bd43f8a245b0a3010a63dbd535f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.610858 1956 tablet_service.cc:1505] Processing CreateTablet for tablet b1ca5b8379c94deab1130bfe8bdc87b0 (DEFAULT_TABLE table=default.Table_85c523c971b34e68bed4afa5d1aac450 [id=32c06e4782a8441c8d1614e81277b52e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.611387 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1ca5b8379c94deab1130bfe8bdc87b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.613185 2646 raft_consensus.cc:493] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.613389 2646 raft_consensus.cc:515] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.614060 2646 leader_election.cc:290] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.632844 1954 tablet_service.cc:1505] Processing CreateTablet for tablet ed836bb7519e4b0290e6547fa57128bc (DEFAULT_TABLE table=default.Table_ec68beb934274468a4f028b3dbaf2cc0 [id=658f4d8e944a4221a722376dc2137896]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.633437 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed836bb7519e4b0290e6547fa57128bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.634373 2381 tablet_bootstrap.cc:654] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.634704 1955 tablet_service.cc:1505] Processing CreateTablet for tablet de8799a5af2b42cc92ccc0195cff98e7 (DEFAULT_TABLE table=default.Table_fc5e859548014044852b49ab2b64bd13 [id=7885bd43f8a245b0a3010a63dbd535f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.634991 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ae08997af74b66b7d6375212eee4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.635908 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9156314f98ac4ed9981f6f13fc585043" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:26.636863 2053 leader_election.cc:343] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.637702 2643 raft_consensus.cc:493] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.637869 2643 raft_consensus.cc:515] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.638522 2643 leader_election.cc:290] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:26.638787 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9156314f98ac4ed9981f6f13fc585043" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:26.639848 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8799a5af2b42cc92ccc0195cff98e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:26.639954 2053 leader_election.cc:343] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.640098 2053 leader_election.cc:304] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.640923 2646 raft_consensus.cc:2749] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.641845 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ec035b1caac450d9ab55e74a061ae38" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:26.642055 2118 raft_consensus.cc:2468] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:26.642889 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ec035b1caac450d9ab55e74a061ae38" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:26.643610 2194 leader_election.cc:343] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.643940 2195 leader_election.cc:304] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.644699 2646 raft_consensus.cc:493] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.644878 2646 raft_consensus.cc:515] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.645556 2646 leader_election.cc:290] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.645859 2643 raft_consensus.cc:2804] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:26.646000 2643 raft_consensus.cc:493] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:26.646090 2643 raft_consensus.cc:3060] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.621304 2378 raft_consensus.cc:359] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.652662 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882cd1680aa140f6a63a961cd4aec25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.652992 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8533f1c191042ee9f2c0b99972fe931" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:26.653589 2053 leader_election.cc:343] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.654983 2643 raft_consensus.cc:515] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.655669 2643 leader_election.cc:290] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:26.656754 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ec035b1caac450d9ab55e74a061ae38" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:26.656982 2118 raft_consensus.cc:3060] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.657469 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8533f1c191042ee9f2c0b99972fe931" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:26.657996 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ec035b1caac450d9ab55e74a061ae38" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:26.658639 2194 leader_election.cc:343] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.658885 2376 ts_tablet_manager.cc:1950] T 3d6690e6bf67454398637d9908b9c652 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:26.659322 2378 raft_consensus.cc:385] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.659451 2378 raft_consensus.cc:740] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.659921 2118 raft_consensus.cc:2468] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:26.660542 2378 consensus_queue.cc:260] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.661293 2378 ts_tablet_manager.cc:1434] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.081s user 0.002s sys 0.000s
W20251026 02:56:26.661631 2053 leader_election.cc:343] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.661783 2053 leader_election.cc:304] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.662647 2646 raft_consensus.cc:2749] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.664269 2661 raft_consensus.cc:493] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.669945 2378 tablet_bootstrap.cc:492] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:26.673869 2377 ts_tablet_manager.cc:1916] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.577854 2372 tablet_replica.cc:333] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:26.675616 2816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.150.60:33729 every 8 connection(s)
I20251026 02:56:26.676857 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e701d4023fee45d18c48997aab3227c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.674996 2661 raft_consensus.cc:515] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.678706 2661 leader_election.cc:290] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.671536 2195 leader_election.cc:304] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.680248 2440 raft_consensus.cc:2804] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:26.680402 2440 raft_consensus.cc:697] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:26.680409 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8adf5c6970be46ed8434fd28404b2166" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:26.680713 2440 consensus_queue.cc:237] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.680855 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8adf5c6970be46ed8434fd28404b2166" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:26.681650 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed836bb7519e4b0290e6547fa57128bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.681689 2631 raft_consensus.cc:493] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.681833 2631 raft_consensus.cc:515] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.682384 2631 leader_election.cc:290] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:26.685644 2485 rpc_server.cc:307] RPC server started. Bound to: 127.1.150.60:33729
I20251026 02:56:26.687961 2485 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/info.pb
I20251026 02:56:26.688266 2377 ts_tablet_manager.cc:1929] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:26.688413 2377 log.cc:1199] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/ea910e98d17340efb816a8aa90f0ba7e
I20251026 02:56:26.689034 1624 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu as pid 2485
I20251026 02:56:26.695174 2643 raft_consensus.cc:493] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.695340 2643 raft_consensus.cc:515] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.696133 2643 leader_election.cc:290] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:26.697049 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "340bf8c78a88413299b61a52f0a578f3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:26.697273 2118 raft_consensus.cc:2468] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:26.697638 2195 leader_election.cc:343] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 7dec83bfb9e946db90c10d526af2dfca
I20251026 02:56:26.697893 2195 leader_election.cc:304] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:26.698155 2643 raft_consensus.cc:2804] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:26.698155 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7177931353164b2aa82c00b2807ccd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.698241 2643 raft_consensus.cc:493] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:26.698307 2643 raft_consensus.cc:3060] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.703557 2643 raft_consensus.cc:515] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.703693 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882fe81da21344ed99ea735fa79f73c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.704228 2643 leader_election.cc:290] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:26.704526 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1ca5b8379c94deab1130bfe8bdc87b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.704617 2375 tablet_bootstrap.cc:492] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.704747 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8799a5af2b42cc92ccc0195cff98e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:26.704896 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "340bf8c78a88413299b61a52f0a578f3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:26.705101 2118 raft_consensus.cc:3060] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.706862 2375 tablet_bootstrap.cc:654] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.709832 2377 ts_tablet_manager.cc:1950] T ea910e98d17340efb816a8aa90f0ba7e P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:26.710623 2372 raft_consensus.cc:2243] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.710968 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3d6690e6bf67454398637d9908b9c652 (table default.Table_e071f7a8a409464d810f12e647a4ff0e [id=c98c6fb4910449b0ad4513fc1c8f6f36]) successfully deleted
I20251026 02:56:26.711154 2372 raft_consensus.cc:2272] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.711853 2372 ts_tablet_manager.cc:1916] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.712100 2376 tablet_replica.cc:333] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:26.714208 2376 raft_consensus.cc:2243] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.714439 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet ea910e98d17340efb816a8aa90f0ba7e (table default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]) successfully deleted
I20251026 02:56:26.714669 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "340bf8c78a88413299b61a52f0a578f3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:26.716711 1915 leader_election.cc:304] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:26.718214 2377 tablet_replica.cc:333] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:26.718410 2377 raft_consensus.cc:2243] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.718580 2377 raft_consensus.cc:2272] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.719074 2377 ts_tablet_manager.cc:1916] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.719300 2376 raft_consensus.cc:2272] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:26.719797 2194 leader_election.cc:343] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.719961 2376 ts_tablet_manager.cc:1916] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.721809 2643 raft_consensus.cc:493] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.721954 2643 raft_consensus.cc:515] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.723438 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 155a23a93358487a9564bb5ac175b4eb
I20251026 02:56:26.723980 1661 catalog_manager.cc:6719] ReplaceTablet: tablet 155a23a93358487a9564bb5ac175b4eb deleted and replaced by tablet 386e1bd4635b4a7e8e7f249b577fd978
I20251026 02:56:26.724267 2661 raft_consensus.cc:2749] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.724488 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "340bf8c78a88413299b61a52f0a578f3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:26.725143 2194 leader_election.cc:343] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.727344 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 155a23a93358487a9564bb5ac175b4eb with delete_type TABLET_DATA_DELETED (replaced by tablet 386e1bd4635b4a7e8e7f249b577fd978) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.721401 2646 raft_consensus.cc:493] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.728869 2646 raft_consensus.cc:515] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.729557 2646 leader_election.cc:290] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.725060 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 155a23a93358487a9564bb5ac175b4eb with delete_type TABLET_DATA_DELETED (replaced by tablet 386e1bd4635b4a7e8e7f249b577fd978) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:26.730453 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "155a23a93358487a9564bb5ac175b4eb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:26.730688 2262 raft_consensus.cc:2468] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:26.728264 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7dec83bfb9e946db90c10d526af2dfca" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:26.731060 2053 leader_election.cc:304] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:26.731365 2646 raft_consensus.cc:2804] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:26.731454 2646 raft_consensus.cc:493] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:26.731565 2646 raft_consensus.cc:3060] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.732451 2758 raft_consensus.cc:493] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.732632 2758 raft_consensus.cc:515] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.733465 2758 leader_election.cc:290] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.734148 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a08e8b7eb1949408f2151f2d7257e9d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:26.734390 2262 raft_consensus.cc:2468] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:26.734993 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 with delete_type TABLET_DATA_DELETED (replaced by tablet 94886ad6d6bb45f39261725169774ac9) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.735302 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 882fe81da21344ed99ea735fa79f73c0
I20251026 02:56:26.735370 2053 leader_election.cc:304] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:26.735641 2758 raft_consensus.cc:2804] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:26.735723 2758 raft_consensus.cc:493] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:26.735823 2758 raft_consensus.cc:3060] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.736517 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet ac19c5a847454a23ab16831198ff2a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:26.737425 2194 leader_election.cc:343] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.737572 2194 leader_election.cc:304] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.738116 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 882fe81da21344ed99ea735fa79f73c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:26.738433 2631 raft_consensus.cc:2749] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.739121 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 882fe81da21344ed99ea735fa79f73c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.739809 2381 tablet_bootstrap.cc:492] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:26.740001 2381 ts_tablet_manager.cc:1403] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.131s user 0.000s sys 0.009s
I20251026 02:56:26.741884 2381 raft_consensus.cc:359] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.743019 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "155a23a93358487a9564bb5ac175b4eb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:26.745626 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet a378e913112c443db869016e3b98fd0e from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:26.746897 2053 leader_election.cc:343] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.748114 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a08e8b7eb1949408f2151f2d7257e9d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:26.748978 2053 leader_election.cc:343] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.750737 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c570e5f9a93c45bda32ec92c68794714" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:26.750980 2117 raft_consensus.cc:2468] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:26.762733 2758 raft_consensus.cc:515] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.751569 2195 leader_election.cc:304] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:26.751740 2643 leader_election.cc:290] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:26.763513 2758 leader_election.cc:290] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.763727 2758 raft_consensus.cc:493] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.763895 2758 raft_consensus.cc:515] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.764657 2381 raft_consensus.cc:385] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.765318 2758 leader_election.cc:290] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.765967 2646 raft_consensus.cc:515] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.766567 2646 leader_election.cc:290] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.767596 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_542a136f8cdd4d32b2e1e6fbf7bcc448" } modify_external_catalogs: true
I20251026 02:56:26.767858 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_542a136f8cdd4d32b2e1e6fbf7bcc448" } modify_external_catalogs: true
I20251026 02:56:26.768515 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_21d71fb70373400396a75b6e52416078" } modify_external_catalogs: true
I20251026 02:56:26.768699 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_21d71fb70373400396a75b6e52416078" } modify_external_catalogs: true
I20251026 02:56:26.768862 2118 raft_consensus.cc:2468] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:26.746609 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 155a23a93358487a9564bb5ac175b4eb with delete_type TABLET_DATA_DELETED (replaced by tablet 386e1bd4635b4a7e8e7f249b577fd978) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.771112 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e2d286228a8244d2bded2ee992c5a064" } new_table_name: "default.Table_2e3225d53ae74a61a8f6e6c91d55f8e1" modify_external_catalogs: true
I20251026 02:56:26.778244 2381 raft_consensus.cc:740] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.764828 2643 raft_consensus.cc:2804] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:26.780313 2643 raft_consensus.cc:493] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:26.780666 2643 raft_consensus.cc:3060] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.785938 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c570e5f9a93c45bda32ec92c68794714" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:26.783756 2195 leader_election.cc:304] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.787621 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a08e8b7eb1949408f2151f2d7257e9d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:26.787840 2262 raft_consensus.cc:3060] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.787899 2261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "155a23a93358487a9564bb5ac175b4eb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:26.788089 2261 raft_consensus.cc:3060] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.790989 2262 raft_consensus.cc:2468] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:26.791077 2261 raft_consensus.cc:2468] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:26.791316 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e5bb0a6a9f947cf9ce119c1fc91fbf5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:26.791898 2053 leader_election.cc:304] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:26.792429 2053 leader_election.cc:304] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:26.792721 2758 raft_consensus.cc:2804] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:26.792886 2758 raft_consensus.cc:697] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:26.792889 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 882fe81da21344ed99ea735fa79f73c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:26.792953 2053 leader_election.cc:343] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.793289 2646 raft_consensus.cc:2804] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:26.793196 2758 consensus_queue.cc:237] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.793833 2440 raft_consensus.cc:2804] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:26.794097 2440 raft_consensus.cc:697] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:26.794348 2440 consensus_queue.cc:237] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.794644 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a08e8b7eb1949408f2151f2d7257e9d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:26.795554 2646 raft_consensus.cc:697] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:26.795840 2646 consensus_queue.cc:237] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:26.796124 2194 leader_election.cc:343] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:26.797017 2053 leader_election.cc:343] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.793322 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e5bb0a6a9f947cf9ce119c1fc91fbf5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:26.801236 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 20dec74440cf4bb9949c462ec51e9889 with delete_type TABLET_DATA_DELETED (replaced by tablet c570e5f9a93c45bda32ec92c68794714) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:26.802385 2053 leader_election.cc:343] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.802580 2053 leader_election.cc:304] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.802906 2826 raft_consensus.cc:2749] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.803236 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "155a23a93358487a9564bb5ac175b4eb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:26.804563 2053 leader_election.cc:343] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.805493 2378 tablet_bootstrap.cc:654] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.806252 2643 raft_consensus.cc:515] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.807363 2381 consensus_queue.cc:260] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.808193 2381 ts_tablet_manager.cc:1434] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.068s user 0.002s sys 0.000s
I20251026 02:56:26.808499 2381 tablet_bootstrap.cc:492] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:26.809613 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_12a3f7cac0614b76888ec0101475ab29" } new_table_name: "default.Table_5f363b68bc2c4e7685f25e4eefa532a8" modify_external_catalogs: true
I20251026 02:56:26.810950 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c570e5f9a93c45bda32ec92c68794714" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:26.811457 2194 leader_election.cc:343] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:26.818251 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de8799a5af2b42cc92ccc0195cff98e7 consensus state has no leader
I20251026 02:56:26.823588 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3dd70def7f1046ea9f2b5bbb7a74e4cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:26.824165 2372 ts_tablet_manager.cc:1929] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251026 02:56:26.824752 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dd70def7f1046ea9f2b5bbb7a74e4cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:26.831156 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e701d4023fee45d18c48997aab3227c2 consensus state has no leader
I20251026 02:56:26.831187 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet e62064fbdffb4954a3a1227efd6327ee
I20251026 02:56:26.831972 2372 log.cc:1199] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/5580ac28a98c4ec4a1d83fb9f9bf25ee
I20251026 02:56:26.832034 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet d65b0d79e2e54af9810022b683c3288b
I20251026 02:56:26.833555 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet e62064fbdffb4954a3a1227efd6327ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:26.833837 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet d65b0d79e2e54af9810022b683c3288b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:26.835125 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet d65b0d79e2e54af9810022b683c3288b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.835317 2372 ts_tablet_manager.cc:1950] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:26.835354 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet e62064fbdffb4954a3a1227efd6327ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.836367 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee (table default.Table_d2dcec579c684b92a92d98fb59ee3da1 [id=4f7edfb6d75c4a0b94ca99bb9cfe14f3]) successfully deleted
W20251026 02:56:26.836717 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7801f073d5d94076af9299492f86fad2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7801f073d5d94076af9299492f86fad2 already in progress: creating tablet
I20251026 02:56:26.837044 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet f85c2745a371455a8dda67f31a19c9d5
I20251026 02:56:26.838353 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet d65b0d79e2e54af9810022b683c3288b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.839074 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet f85c2745a371455a8dda67f31a19c9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.839435 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet f85c2745a371455a8dda67f31a19c9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:26.840044 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet e62064fbdffb4954a3a1227efd6327ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:26.840101 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet f85c2745a371455a8dda67f31a19c9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:26.841471 2440 raft_consensus.cc:493] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.841640 2440 raft_consensus.cc:515] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.842356 2440 leader_election.cc:290] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:26.844496 2372 tablet_replica.cc:333] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:26.844743 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a5343ba771a4e8ea2f467c4c619c1f9" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:26.845094 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a70d70abc3a34dac8cf9a747903fa78c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a70d70abc3a34dac8cf9a747903fa78c already in progress: creating tablet
I20251026 02:56:26.845281 2376 ts_tablet_manager.cc:1929] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251026 02:56:26.845328 2194 leader_election.cc:343] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.846439 2372 raft_consensus.cc:2243] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.846659 2372 raft_consensus.cc:2272] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.846024 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a5343ba771a4e8ea2f467c4c619c1f9" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:26.848805 2118 raft_consensus.cc:2468] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:26.849278 2195 leader_election.cc:304] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.849278 2308 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:26.849674 2308 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:26.849689 2372 ts_tablet_manager.cc:1916] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.849731 2440 raft_consensus.cc:2804] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:26.849848 2440 raft_consensus.cc:493] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:26.849958 2440 raft_consensus.cc:3060] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.855029 2440 raft_consensus.cc:515] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.856988 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a5343ba771a4e8ea2f467c4c619c1f9" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:26.857301 2117 raft_consensus.cc:3060] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.860577 2440 leader_election.cc:290] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:26.862808 2117 raft_consensus.cc:2468] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:26.863569 2195 leader_election.cc:304] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:26.864480 2440 raft_consensus.cc:2804] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:26.864856 2440 raft_consensus.cc:697] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:26.865249 2440 consensus_queue.cc:237] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.870787 2372 ts_tablet_manager.cc:1929] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:26.872432 2308 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:26.872611 2372 log.cc:1199] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/8667ffa25f2b478a987826d486400bbe
I20251026 02:56:26.873418 2372 ts_tablet_manager.cc:1950] T 8667ffa25f2b478a987826d486400bbe P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:26.876940 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 046a34f9408b406ba08471421f422064 consensus state has no leader
I20251026 02:56:26.877549 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 8667ffa25f2b478a987826d486400bbe (table default.Table_4fd5f39751624015ad57dd7fc77ce18a [id=f7f1e852a46f4129b2709fd58a6be034]) successfully deleted
I20251026 02:56:26.878263 2372 tablet_replica.cc:333] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:26.879037 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_528af198a403496a8c1b27fa5821123b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:26.879279 2372 raft_consensus.cc:2243] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.879535 2372 raft_consensus.cc:2272] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:26.879820 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_528af198a403496a8c1b27fa5821123b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.880153 2372 ts_tablet_manager.cc:1916] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.880232 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8b019eeaa6074ffb87c0935572a36ed7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.880690 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b019eeaa6074ffb87c0935572a36ed7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.881770 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e2223a0634ec4d9c96a847247e20a762" } new_table_name: "default.Table_3792bcbbc16b4be0b113cb5f07b9261b" modify_external_catalogs: true
I20251026 02:56:26.883173 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8a66b2dca58445b1877938b41d7b4507"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.883805 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a66b2dca58445b1877938b41d7b4507 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.884774 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f439acecb4bb4404a14e11f5824c654b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.885345 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f439acecb4bb4404a14e11f5824c654b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.887082 2372 ts_tablet_manager.cc:1929] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:26.892058 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a5343ba771a4e8ea2f467c4c619c1f9" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:26.893323 2376 log.cc:1199] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/ea910e98d17340efb816a8aa90f0ba7e
I20251026 02:56:26.893821 2376 ts_tablet_manager.cc:1950] T ea910e98d17340efb816a8aa90f0ba7e P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:26.894335 2027 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:26.894768 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:26.895637 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet ea910e98d17340efb816a8aa90f0ba7e (table default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]) successfully deleted
I20251026 02:56:26.897893 2376 tablet_replica.cc:333] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:26.907055 2376 raft_consensus.cc:2243] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:26.901120 2375 tablet_bootstrap.cc:492] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.907600 2375 ts_tablet_manager.cc:1403] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.203s user 0.040s sys 0.048s
I20251026 02:56:26.908493 2375 raft_consensus.cc:359] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.908703 2375 raft_consensus.cc:385] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.908767 2375 raft_consensus.cc:740] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.909005 2375 consensus_queue.cc:260] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.909674 2375 ts_tablet_manager.cc:1434] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:26.909819 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c570e5f9a93c45bda32ec92c68794714" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:26.910089 2118 raft_consensus.cc:3060] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:26.910416 2375 tablet_bootstrap.cc:492] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.912546 2375 tablet_bootstrap.cc:654] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.914731 2824 raft_consensus.cc:493] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.914969 2824 raft_consensus.cc:515] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:26.892818 2194 leader_election.cc:343] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.917740 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84ad6c72a8704c148411881760848df4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:26.915879 2824 leader_election.cc:290] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:26.918606 2759 consensus_queue.cc:1048] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:26.920826 2053 leader_election.cc:343] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.921046 2376 raft_consensus.cc:2272] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.912722 2378 tablet_bootstrap.cc:492] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:26.916874 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84ad6c72a8704c148411881760848df4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:26.920194 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:26.921592 2372 log.cc:1199] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/a2bfbe624075463f8b90ed4ec66b784c
W20251026 02:56:26.921734 2053 leader_election.cc:343] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.921744 2378 ts_tablet_manager.cc:1403] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.252s user 0.006s sys 0.006s
I20251026 02:56:26.921880 2053 leader_election.cc:304] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.922125 2372 ts_tablet_manager.cc:1950] T a2bfbe624075463f8b90ed4ec66b784c P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:26.922767 2378 raft_consensus.cc:359] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.922976 2378 raft_consensus.cc:385] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.923233 2372 tablet_replica.cc:333] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:26.923416 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_e2223a0634ec4d9c96a847247e20a762 [id=8a3d1c20763445c499bd655eb99deb56]) successfully deleted
I20251026 02:56:26.923530 2372 raft_consensus.cc:2243] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:26.923748 2378 raft_consensus.cc:740] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.923837 2372 pending_rounds.cc:70] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48: Trying to abort 1 pending ops.
I20251026 02:56:26.923964 2372 pending_rounds.cc:77] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888496318271488 op_type: NO_OP noop_request { }
I20251026 02:56:26.924122 2372 raft_consensus.cc:2889] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:26.924288 2372 raft_consensus.cc:2272] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:26.924348 2378 consensus_queue.cc:260] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.924788 2824 raft_consensus.cc:2749] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.925591 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6d91747dd6164f0d84509497a93aeeea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.926294 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d91747dd6164f0d84509497a93aeeea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.930203 2378 ts_tablet_manager.cc:1434] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251026 02:56:26.930562 2378 tablet_bootstrap.cc:492] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:26.932277 2165 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:26.932570 2165 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:26.936980 2372 ts_tablet_manager.cc:1916] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.937036 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a1e745ce0f454f4fa2732de1282afae1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:26.937747 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1e745ce0f454f4fa2732de1282afae1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:26.941210 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet ae95654d129441e08fcb3f3b355ffc22 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:26.941783 2118 raft_consensus.cc:2468] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:26.942440 2195 leader_election.cc:304] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.942826 2759 raft_consensus.cc:2804] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:26.943010 2759 raft_consensus.cc:697] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:26.943360 2759 consensus_queue.cc:237] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.952790 2165 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
W20251026 02:56:26.957667 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec13c5386abe414ba25589e03d5b850a has no consensus state
I20251026 02:56:26.958039 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 5caeb11322fe48348ebb523967152187 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:26.958688 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 6a3491b303f94fe2a2c838297f73d464 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:26.961130 2372 ts_tablet_manager.cc:1929] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:26.964121 2372 log.cc:1199] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/155a23a93358487a9564bb5ac175b4eb
I20251026 02:56:26.964792 2372 ts_tablet_manager.cc:1950] T 155a23a93358487a9564bb5ac175b4eb P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:26.965931 2376 ts_tablet_manager.cc:1916] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:26.966150 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 155a23a93358487a9564bb5ac175b4eb (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]) successfully deleted
W20251026 02:56:26.969026 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 882fe81da21344ed99ea735fa79f73c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 882fe81da21344ed99ea735fa79f73c0 already in progress: creating tablet
I20251026 02:56:26.898679 2381 tablet_bootstrap.cc:654] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.899466 2440 raft_consensus.cc:493] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.905828 2631 raft_consensus.cc:493] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:26.969805 2631 raft_consensus.cc:515] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:26.970013 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet e62064fbdffb4954a3a1227efd6327ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e62064fbdffb4954a3a1227efd6327ee already in progress: creating tablet
W20251026 02:56:26.970620 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet d65b0d79e2e54af9810022b683c3288b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d65b0d79e2e54af9810022b683c3288b already in progress: creating tablet
I20251026 02:56:26.970659 2631 leader_election.cc:290] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:26.971089 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac19c5a847454a23ab16831198ff2a20" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:26.971206 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet f85c2745a371455a8dda67f31a19c9d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f85c2745a371455a8dda67f31a19c9d5 already in progress: creating tablet
I20251026 02:56:26.971280 2440 raft_consensus.cc:515] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.972302 2819 sys_catalog.cc:263] Verifying existing consensus state
W20251026 02:56:26.976848 2194 leader_election.cc:343] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:26.977015 2195 leader_election.cc:343] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 5580ac28a98c4ec4a1d83fb9f9bf25ee
I20251026 02:56:26.977200 2440 leader_election.cc:290] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:26.978121 2195 leader_election.cc:343] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ac19c5a847454a23ab16831198ff2a20
I20251026 02:56:26.978281 2195 leader_election.cc:304] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.978754 2440 raft_consensus.cc:2749] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.980818 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5580ac28a98c4ec4a1d83fb9f9bf25ee" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:26.981477 2194 leader_election.cc:343] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:26.981618 2194 leader_election.cc:304] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:26.982609 2378 tablet_bootstrap.cc:654] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.982890 2440 raft_consensus.cc:2749] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:26.984297 2308 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:26.984522 2308 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:26.988967 2819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Bootstrap starting.
I20251026 02:56:26.990125 2375 tablet_bootstrap.cc:492] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:26.990314 2375 ts_tablet_manager.cc:1403] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.080s user 0.017s sys 0.008s
I20251026 02:56:26.991246 2375 raft_consensus.cc:359] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.991464 2375 raft_consensus.cc:385] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:26.991566 2375 raft_consensus.cc:740] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:26.991986 2375 consensus_queue.cc:260] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:26.992630 2165 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:26.992728 2375 ts_tablet_manager.cc:1434] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:26.993214 2375 tablet_bootstrap.cc:492] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:26.995141 2375 tablet_bootstrap.cc:654] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:26.999863 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_09ca38422af543f89eab26b4117536dc" } modify_external_catalogs: true
I20251026 02:56:27.000082 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_09ca38422af543f89eab26b4117536dc" } modify_external_catalogs: true
I20251026 02:56:27.003841 2375 tablet_bootstrap.cc:492] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:27.004017 2375 ts_tablet_manager.cc:1403] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20251026 02:56:27.004864 2375 raft_consensus.cc:359] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.006877 2375 raft_consensus.cc:385] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.006985 2375 raft_consensus.cc:740] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.007234 2375 consensus_queue.cc:260] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.007750 2375 ts_tablet_manager.cc:1434] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251026 02:56:27.008078 2375 tablet_bootstrap.cc:492] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:27.009253 1954 tablet_service.cc:1505] Processing CreateTablet for tablet bfa68f2c942c43e2a02e9d333bdfa2a2 (DEFAULT_TABLE table=default.Table_5ad87a5fcdd745529b77d2ffd8c2a4ff [id=84eea78390b0408dbfeda88cafa9dcd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.009683 2375 tablet_bootstrap.cc:654] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:27.009804 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa68f2c942c43e2a02e9d333bdfa2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.012442 2165 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.013562 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 340bf8c78a88413299b61a52f0a578f3 (table default.Table_1361fbf89aa04321a7876c1e43e27089 [id=c162ef89d2f14e91a24a6d8c2c37c261]). Expected version 4 got 0
I20251026 02:56:27.013856 1661 catalog_manager.cc:5649] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:27.014304 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 903e65fa8c214dc9a50cb97cab101d22 (table default.Table_08b776b4561f401880b0c7ba0a9b6fa2 [id=65b5770c4b9243b892bfab963a439bb9]). Expected version 2 got 0
I20251026 02:56:27.014595 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 9a08e8b7eb1949408f2151f2d7257e9d (table default.Table_12a3f7cac0614b76888ec0101475ab29 [id=43cb7f538f824ae38059645230cc7e94]). Expected version 2 got 0
I20251026 02:56:27.014781 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet a821ed8a686c452a975acb96b5c74778 (table default.Table_00dc4c49a19246889353df2157eebf15 [id=347d712aca4e477c89805846800fb824]). Expected version 1 got 0
I20251026 02:56:27.015010 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet c570e5f9a93c45bda32ec92c68794714 (table default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]). Expected version 3 got 0
I20251026 02:56:27.022903 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 21b936fa4f6a4ff6af321971a30cccbe (DEFAULT_TABLE table=default.Table_1bc1610289a448d99b780c0e289b7d58 [id=5fb61e88f7074976bac29b40f4466ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.023183 2084 tablet_service.cc:1505] Processing CreateTablet for tablet d212170f056a44dfb15cfcb7f9fbae2f (DEFAULT_TABLE table=default.Table_15e626aa074c4736a61a03be25b64350 [id=822eea61b89846f4828511c58a9da4c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.023350 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21b936fa4f6a4ff6af321971a30cccbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.023645 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d212170f056a44dfb15cfcb7f9fbae2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.023970 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 2fd8f02126b24285bbfa575540ba0f09 (DEFAULT_TABLE table=default.Table_61b2d44fec564f0a8b01015ef7ee44ad [id=1950d7cb826b4a99b1ba8866e1d845f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.024397 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fd8f02126b24285bbfa575540ba0f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.024677 2086 tablet_service.cc:1505] Processing CreateTablet for tablet bfa68f2c942c43e2a02e9d333bdfa2a2 (DEFAULT_TABLE table=default.Table_5ad87a5fcdd745529b77d2ffd8c2a4ff [id=84eea78390b0408dbfeda88cafa9dcd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.025065 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa68f2c942c43e2a02e9d333bdfa2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.031908 2238 tablet_service.cc:1505] Processing CreateTablet for tablet bfa68f2c942c43e2a02e9d333bdfa2a2 (DEFAULT_TABLE table=default.Table_5ad87a5fcdd745529b77d2ffd8c2a4ff [id=84eea78390b0408dbfeda88cafa9dcd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.032179 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 21b936fa4f6a4ff6af321971a30cccbe (DEFAULT_TABLE table=default.Table_1bc1610289a448d99b780c0e289b7d58 [id=5fb61e88f7074976bac29b40f4466ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.032454 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 2fd8f02126b24285bbfa575540ba0f09 (DEFAULT_TABLE table=default.Table_61b2d44fec564f0a8b01015ef7ee44ad [id=1950d7cb826b4a99b1ba8866e1d845f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.032611 2235 tablet_service.cc:1505] Processing CreateTablet for tablet d212170f056a44dfb15cfcb7f9fbae2f (DEFAULT_TABLE table=default.Table_15e626aa074c4736a61a03be25b64350 [id=822eea61b89846f4828511c58a9da4c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.031912 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 3e38f282e8414b7db2a6d5f4a8933e84 (DEFAULT_TABLE table=default.Table_81e9f54ed3934b6b80f7c10d0be54d9a [id=e2f6dfa3686b41ddbc9f3b0205c7f05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.033813 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 21b936fa4f6a4ff6af321971a30cccbe (DEFAULT_TABLE table=default.Table_1bc1610289a448d99b780c0e289b7d58 [id=5fb61e88f7074976bac29b40f4466ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.034301 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21b936fa4f6a4ff6af321971a30cccbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.037424 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 2fd8f02126b24285bbfa575540ba0f09 (DEFAULT_TABLE table=default.Table_61b2d44fec564f0a8b01015ef7ee44ad [id=1950d7cb826b4a99b1ba8866e1d845f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.038239 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fd8f02126b24285bbfa575540ba0f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.040663 1960 tablet_service.cc:1505] Processing CreateTablet for tablet d212170f056a44dfb15cfcb7f9fbae2f (DEFAULT_TABLE table=default.Table_15e626aa074c4736a61a03be25b64350 [id=822eea61b89846f4828511c58a9da4c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.041168 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d212170f056a44dfb15cfcb7f9fbae2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.041335 2819 log.cc:826] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Log is configured to *not* fsync() on all Append() calls
I20251026 02:56:27.052609 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 3e38f282e8414b7db2a6d5f4a8933e84 (DEFAULT_TABLE table=default.Table_81e9f54ed3934b6b80f7c10d0be54d9a [id=e2f6dfa3686b41ddbc9f3b0205c7f05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.053189 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e38f282e8414b7db2a6d5f4a8933e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.055238 2381 tablet_bootstrap.cc:492] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:27.055431 2381 ts_tablet_manager.cc:1403] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.247s user 0.002s sys 0.003s
I20251026 02:56:27.056316 2381 raft_consensus.cc:359] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.056548 2381 raft_consensus.cc:385] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.056635 2381 raft_consensus.cc:740] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.057006 2381 consensus_queue.cc:260] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.057977 2243 tablet_service.cc:1505] Processing CreateTablet for tablet c31d2704d1f14ddead69c794cc631ec1 (DEFAULT_TABLE table=default.Table_ad7f0263e7ad4bdea1b9805522ec23db [id=9b9d43027de84287b1e2fd35f5e4d5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.058507 2096 tablet_service.cc:1505] Processing CreateTablet for tablet c31d2704d1f14ddead69c794cc631ec1 (DEFAULT_TABLE table=default.Table_ad7f0263e7ad4bdea1b9805522ec23db [id=9b9d43027de84287b1e2fd35f5e4d5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.059831 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31d2704d1f14ddead69c794cc631ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.060462 2231 tablet_service.cc:1505] Processing CreateTablet for tablet f437d8b5f5784e8fbec10d0700e90348 (DEFAULT_TABLE table=default.Table_b555f404492a47808d8a9a168a45ef2f [id=6185bafc43f340dfa6e187a6e5468874]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:26.908705 2377 ts_tablet_manager.cc:1929] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:26.909185 2643 leader_election.cc:290] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:27.061528 2241 tablet_service.cc:1505] Processing CreateTablet for tablet ec13c5386abe414ba25589e03d5b850a (DEFAULT_TABLE table=default.Table_3792bcbbc16b4be0b113cb5f07b9261b [id=8a3d1c20763445c499bd655eb99deb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.061887 2377 log.cc:1199] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/0020cb1f190c49978c5322d387a47f71
I20251026 02:56:27.062546 2377 ts_tablet_manager.cc:1950] T 0020cb1f190c49978c5322d387a47f71 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:27.065399 2240 tablet_service.cc:1505] Processing CreateTablet for tablet fe48472efa444bf580e4074c96a9de92 (DEFAULT_TABLE table=default.Table_7488ff00e4c04293b85e265c4d686fc2 [id=7e3d19f1d75d4ebb94c220327ff8abcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.067831 2236 tablet_service.cc:1505] Processing CreateTablet for tablet aa10cd990a944eab8d9eb66aaf20080c (DEFAULT_TABLE table=default.Table_42c4c5e055054536b2b541711288a481 [id=503128cdfe654cfb9713c8e241eac740]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.070030 1658 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 1d8cdf79a97149618bcaf0fefee1dfeb (table default.Table_424693529d0d4c829755adc2c9835c0e [id=690ec15818d34cdfa990c8fcd36a19cd]). Expected version 1 got 0
I20251026 02:56:27.071187 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3d25c519f16647ca962f3862c6cd4f98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.071851 1961 tablet_service.cc:1505] Processing CreateTablet for tablet c31d2704d1f14ddead69c794cc631ec1 (DEFAULT_TABLE table=default.Table_ad7f0263e7ad4bdea1b9805522ec23db [id=9b9d43027de84287b1e2fd35f5e4d5f5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:27.072474 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d25c519f16647ca962f3862c6cd4f98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.072754 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.075733 1951 tablet_service.cc:1505] Processing CreateTablet for tablet f437d8b5f5784e8fbec10d0700e90348 (DEFAULT_TABLE table=default.Table_b555f404492a47808d8a9a168a45ef2f [id=6185bafc43f340dfa6e187a6e5468874]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.075865 2088 tablet_service.cc:1505] Processing CreateTablet for tablet ec13c5386abe414ba25589e03d5b850a (DEFAULT_TABLE table=default.Table_3792bcbbc16b4be0b113cb5f07b9261b [id=8a3d1c20763445c499bd655eb99deb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.076313 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec13c5386abe414ba25589e03d5b850a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.077682 2091 tablet_service.cc:1505] Processing CreateTablet for tablet fe48472efa444bf580e4074c96a9de92 (DEFAULT_TABLE table=default.Table_7488ff00e4c04293b85e265c4d686fc2 [id=7e3d19f1d75d4ebb94c220327ff8abcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.078078 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe48472efa444bf580e4074c96a9de92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.079062 2087 tablet_service.cc:1505] Processing CreateTablet for tablet f437d8b5f5784e8fbec10d0700e90348 (DEFAULT_TABLE table=default.Table_b555f404492a47808d8a9a168a45ef2f [id=6185bafc43f340dfa6e187a6e5468874]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.079756 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f437d8b5f5784e8fbec10d0700e90348. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.080672 2381 ts_tablet_manager.cc:1434] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20251026 02:56:27.082530 1955 tablet_service.cc:1505] Processing CreateTablet for tablet ec13c5386abe414ba25589e03d5b850a (DEFAULT_TABLE table=default.Table_3792bcbbc16b4be0b113cb5f07b9261b [id=8a3d1c20763445c499bd655eb99deb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.083885 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31d2704d1f14ddead69c794cc631ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.083868 2098 tablet_service.cc:1505] Processing CreateTablet for tablet aa10cd990a944eab8d9eb66aaf20080c (DEFAULT_TABLE table=default.Table_42c4c5e055054536b2b541711288a481 [id=503128cdfe654cfb9713c8e241eac740]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.084278 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa10cd990a944eab8d9eb66aaf20080c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.085351 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 3e38f282e8414b7db2a6d5f4a8933e84 (DEFAULT_TABLE table=default.Table_81e9f54ed3934b6b80f7c10d0be54d9a [id=e2f6dfa3686b41ddbc9f3b0205c7f05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.085875 1979 raft_consensus.cc:3060] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:27.085223 1962 tablet_service.cc:1505] Processing CreateTablet for tablet aa10cd990a944eab8d9eb66aaf20080c (DEFAULT_TABLE table=default.Table_42c4c5e055054536b2b541711288a481 [id=503128cdfe654cfb9713c8e241eac740]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.088138 1956 tablet_service.cc:1505] Processing CreateTablet for tablet fe48472efa444bf580e4074c96a9de92 (DEFAULT_TABLE table=default.Table_7488ff00e4c04293b85e265c4d686fc2 [id=7e3d19f1d75d4ebb94c220327ff8abcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.090818 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec13c5386abe414ba25589e03d5b850a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.091620 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f437d8b5f5784e8fbec10d0700e90348. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.092199 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe48472efa444bf580e4074c96a9de92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.096402 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa68f2c942c43e2a02e9d333bdfa2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.096686 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d212170f056a44dfb15cfcb7f9fbae2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.097270 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa10cd990a944eab8d9eb66aaf20080c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.097450 2643 consensus_queue.cc:1048] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:27.098053 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e38f282e8414b7db2a6d5f4a8933e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.100432 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21b936fa4f6a4ff6af321971a30cccbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.104105 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fd8f02126b24285bbfa575540ba0f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.104357 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e38f282e8414b7db2a6d5f4a8933e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.111272 2661 raft_consensus.cc:493] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:27.111507 2661 raft_consensus.cc:515] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:27.113462 1915 leader_election.cc:343] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 20dec74440cf4bb9949c462ec51e9889
I20251026 02:56:27.113765 2643 raft_consensus.cc:493] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:27.113940 2643 raft_consensus.cc:515] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.114614 2643 leader_election.cc:290] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:27.115531 2661 leader_election.cc:290] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:27.119731 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31d2704d1f14ddead69c794cc631ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.123639 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f437d8b5f5784e8fbec10d0700e90348. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.124577 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec13c5386abe414ba25589e03d5b850a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.127552 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ae95654d129441e08fcb3f3b355ffc22
I20251026 02:56:27.128142 1657 catalog_manager.cc:6719] ReplaceTablet: tablet ae95654d129441e08fcb3f3b355ffc22 deleted and replaced by tablet c2d6691bb4cf4a8292b0c082e54a91a5
I20251026 02:56:27.128995 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_62fd5a902d0648cea484f259d02de5f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.129580 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62fd5a902d0648cea484f259d02de5f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:27.134119 1916 leader_election.cc:343] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 20dec74440cf4bb9949c462ec51e9889
I20251026 02:56:27.134141 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0f53c2a8d63c47769dca3283cb871b9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.134291 1916 leader_election.cc:304] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
W20251026 02:56:27.134668 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f53c2a8d63c47769dca3283cb871b9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.134879 2661 raft_consensus.cc:2749] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:27.136844 2375 tablet_bootstrap.cc:492] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:27.137074 2375 ts_tablet_manager.cc:1403] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.129s user 0.024s sys 0.001s
I20251026 02:56:27.137650 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ca4611c0a494425ca50d599b310872c8
I20251026 02:56:27.137719 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a821ed8a686c452a975acb96b5c74778" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:27.138325 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a821ed8a686c452a975acb96b5c74778" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:27.138551 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_61b2d44fec564f0a8b01015ef7ee44ad" } new_table_name: "default.Table_7539d36a088f4eafaa21d151ed9ea1cb" modify_external_catalogs: true
I20251026 02:56:27.139251 2118 raft_consensus.cc:2468] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:27.140779 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet ae95654d129441e08fcb3f3b355ffc22 with delete_type TABLET_DATA_DELETED (replaced by tablet c2d6691bb4cf4a8292b0c082e54a91a5) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.141461 2089 tablet_service.cc:1552] Processing DeleteTablet for tablet ca4611c0a494425ca50d599b310872c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.144094 2378 tablet_bootstrap.cc:492] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:27.144469 2378 ts_tablet_manager.cc:1403] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.214s user 0.011s sys 0.007s
I20251026 02:56:27.145372 2378 raft_consensus.cc:359] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.145710 2378 raft_consensus.cc:385] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.145916 2378 raft_consensus.cc:740] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.146402 2378 consensus_queue.cc:260] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.146888 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet ca4611c0a494425ca50d599b310872c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.147351 2378 ts_tablet_manager.cc:1434] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:27.147897 2378 tablet_bootstrap.cc:492] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:27.148511 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ae95654d129441e08fcb3f3b355ffc22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae95654d129441e08fcb3f3b355ffc22 already in progress: creating tablet
W20251026 02:56:27.148805 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ca4611c0a494425ca50d599b310872c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca4611c0a494425ca50d599b310872c8 already in progress: creating tablet
I20251026 02:56:27.149154 2375 raft_consensus.cc:359] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.149407 2375 raft_consensus.cc:385] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.150404 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe48472efa444bf580e4074c96a9de92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.150941 2375 raft_consensus.cc:740] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.151266 2375 consensus_queue.cc:260] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.150440 2381 tablet_bootstrap.cc:492] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:27.152444 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa10cd990a944eab8d9eb66aaf20080c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.152554 2375 ts_tablet_manager.cc:1434] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251026 02:56:27.152882 2375 tablet_bootstrap.cc:492] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:27.154306 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet ca4611c0a494425ca50d599b310872c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.160601 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 5caeb11322fe48348ebb523967152187
I20251026 02:56:27.161127 1663 catalog_manager.cc:6719] ReplaceTablet: tablet 5caeb11322fe48348ebb523967152187 deleted and replaced by tablet d43b6f22dd4f4f9c9657f21514550312
I20251026 02:56:27.162317 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 6a3491b303f94fe2a2c838297f73d464
I20251026 02:56:27.162788 1654 catalog_manager.cc:6719] ReplaceTablet: tablet 6a3491b303f94fe2a2c838297f73d464 deleted and replaced by tablet 4b5e61afe90e4bc4a2897de21e0b3919
I20251026 02:56:27.165084 1660 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet c570e5f9a93c45bda32ec92c68794714 (table default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]). Expected version 3 got 0
I20251026 02:56:27.165338 1660 catalog_manager.cc:5649] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:27.165741 1660 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]). Expected version 1 got 0
I20251026 02:56:27.165944 1660 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 903e65fa8c214dc9a50cb97cab101d22 (table default.Table_08b776b4561f401880b0c7ba0a9b6fa2 [id=65b5770c4b9243b892bfab963a439bb9]). Expected version 2 got 0
I20251026 02:56:27.166081 1660 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 340bf8c78a88413299b61a52f0a578f3 (table default.Table_1361fbf89aa04321a7876c1e43e27089 [id=c162ef89d2f14e91a24a6d8c2c37c261]). Expected version 4 got 0
I20251026 02:56:27.166371 1660 catalog_manager.cc:5649] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:27.166828 1660 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (table default.Table_fa634331e4bd4c14a102908bc41edba1 [id=1d09f5b9fa164ac783f0be9332eac4c3]). Expected version 4 got 0
I20251026 02:56:27.167076 1660 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 9a08e8b7eb1949408f2151f2d7257e9d (table default.Table_12a3f7cac0614b76888ec0101475ab29 [id=43cb7f538f824ae38059645230cc7e94]). Expected version 2 got 0
I20251026 02:56:27.168197 2089 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a3491b303f94fe2a2c838297f73d464 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b5e61afe90e4bc4a2897de21e0b3919) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.169075 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 5caeb11322fe48348ebb523967152187 with delete_type TABLET_DATA_DELETED (replaced by tablet d43b6f22dd4f4f9c9657f21514550312) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:27.169199 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 6a3491b303f94fe2a2c838297f73d464 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a3491b303f94fe2a2c838297f73d464 already in progress: creating tablet
W20251026 02:56:27.169636 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a08e8b7eb1949408f2151f2d7257e9d consensus state has no leader
W20251026 02:56:27.170641 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 5caeb11322fe48348ebb523967152187 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5caeb11322fe48348ebb523967152187 already in progress: creating tablet
W20251026 02:56:27.171936 1661 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet c570e5f9a93c45bda32ec92c68794714 (table default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet c570e5f9a93c45bda32ec92c68794714 consensus state has no leader
I20251026 02:56:27.172695 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 81530bbf4a2d4e908add683475e594fb from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:27.178046 2819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=47 overwritten=1 applied=45 ignored=0} inserts{seen=45 ignored=0} mutations{seen=31 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 02:56:27.178654 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet ae95654d129441e08fcb3f3b355ffc22 with delete_type TABLET_DATA_DELETED (replaced by tablet c2d6691bb4cf4a8292b0c082e54a91a5) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.179100 2819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Bootstrap complete.
I20251026 02:56:27.181597 2759 raft_consensus.cc:493] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:27.181779 2440 raft_consensus.cc:493] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:27.181890 2440 raft_consensus.cc:515] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.181460 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet ae95654d129441e08fcb3f3b355ffc22 with delete_type TABLET_DATA_DELETED (replaced by tablet c2d6691bb4cf4a8292b0c082e54a91a5) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.183691 2195 leader_election.cc:304] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:27.185530 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 289) took 1059 ms. Trace:
I20251026 02:56:27.185950 2631 raft_consensus.cc:2804] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:27.186056 2631 raft_consensus.cc:493] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:27.186446 2631 raft_consensus.cc:3060] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:27.189419 2194 leader_election.cc:343] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.191368 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet 5caeb11322fe48348ebb523967152187 with delete_type TABLET_DATA_DELETED (replaced by tablet d43b6f22dd4f4f9c9657f21514550312) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.191529 2759 raft_consensus.cc:515] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.192516 2377 rpcz_store.cc:276] 1026 02:56:26.119210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.120319 (+ 1109us) service_pool.cc:225] Handling call
1026 02:56:26.718119 (+597800us) ts_tablet_manager.cc:1151] Deleting tablet 0020cb1f190c49978c5322d387a47f71
1026 02:56:26.718130 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:26.902728 (+184598us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:26.908526 (+ 5798us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:27.178974 (+270448us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":597534}
I20251026 02:56:27.193038 2377 tablet_replica.cc:333] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:27.194344 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 0020cb1f190c49978c5322d387a47f71 (table default.Table_689f802c9e4b457eb71fb78898fe14b0 [id=05242557de9a4a19948b8591ce73ab28]) successfully deleted
I20251026 02:56:27.194792 2377 raft_consensus.cc:2243] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:27.195037 2377 raft_consensus.cc:2272] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:27.198910 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet 2524ee0c964345459c3e62f1e39b8f9e from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:27.200382 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fd8f02126b24285bbfa575540ba0f09 consensus state has no leader
I20251026 02:56:27.200510 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 47037db3602246a2ab6224557286162f
I20251026 02:56:27.203327 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_241789d6111d497c89efdb4507dff32e" } modify_external_catalogs: true
I20251026 02:56:27.203552 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_241789d6111d497c89efdb4507dff32e" } modify_external_catalogs: true
I20251026 02:56:27.204442 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_08b776b4561f401880b0c7ba0a9b6fa2" } new_table_name: "default.Table_862bd13f099645e8aafd987e4e850ae6" modify_external_catalogs: true
I20251026 02:56:27.204559 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7be811f036924663898ad10ce19955b1" } new_table_name: "default.Table_1fa437ef7f89450bb6f672a1fb491baa" modify_external_catalogs: true
I20251026 02:56:27.206638 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0a8522a9059f4189b8cb07ff7409bd3a" } modify_external_catalogs: true
I20251026 02:56:27.206859 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0a8522a9059f4189b8cb07ff7409bd3a" } modify_external_catalogs: true
I20251026 02:56:27.207458 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0d7b9434a0494a618c25ceebd7f65de8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.207562 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_672f95f49551403a99355e82b17eaece"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.208034 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d7b9434a0494a618c25ceebd7f65de8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:27.208264 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_672f95f49551403a99355e82b17eaece in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.210146 2377 ts_tablet_manager.cc:1916] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:27.212952 2440 leader_election.cc:290] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:27.212037 2819 raft_consensus.cc:359] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:27.219874 2759 leader_election.cc:290] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:27.220647 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet 47037db3602246a2ab6224557286162f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.223460 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a3491b303f94fe2a2c838297f73d464 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b5e61afe90e4bc4a2897de21e0b3919) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.225054 2381 tablet_bootstrap.cc:654] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:27.227140 2308 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.229209 2819 raft_consensus.cc:740] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751b719b75ce4bc2a88a19091230ecb7, State: Initialized, Role: FOLLOWER
W20251026 02:56:27.231523 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 consensus state has no leader
I20251026 02:56:27.233717 2819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 3.46, Last appended by leader: 46, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:27.241870 2849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:27.241959 2631 raft_consensus.cc:515] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.246068 2849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:27.245097 2819 sys_catalog.cc:565] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 02:56:27.249984 2661 raft_consensus.cc:493] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:27.250941 2661 raft_consensus.cc:515] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.251677 2661 leader_election.cc:290] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:27.254024 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 5caeb11322fe48348ebb523967152187 with delete_type TABLET_DATA_DELETED (replaced by tablet d43b6f22dd4f4f9c9657f21514550312) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.255671 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 47037db3602246a2ab6224557286162f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.255827 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a3491b303f94fe2a2c838297f73d464 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b5e61afe90e4bc4a2897de21e0b3919) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.256943 2378 tablet_bootstrap.cc:654] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:27.257822 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8dbc12d49b094dbeadad2e76916ba9af" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:27.259295 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 7176cbe8f64546b7871d047c0e0e5b2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.260130 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a821ed8a686c452a975acb96b5c74778" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:27.260284 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "740f7c7f186f45ceaa4c40e76419e3a1" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:27.265492 2376 ts_tablet_manager.cc:1929] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:27.265650 2376 log.cc:1199] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/d8ae08997af74b66b7d6375212eee4c2
I20251026 02:56:27.267583 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ced7ad6a601d432cbc13b3199a0506e4
I20251026 02:56:27.268860 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet c570e5f9a93c45bda32ec92c68794714
I20251026 02:56:27.269320 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet ced7ad6a601d432cbc13b3199a0506e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.270105 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet c570e5f9a93c45bda32ec92c68794714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.272429 2376 ts_tablet_manager.cc:1950] T d8ae08997af74b66b7d6375212eee4c2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:27.273829 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet d8ae08997af74b66b7d6375212eee4c2 (table default.Table_c095574955cf44d7aecfbca77de4ad1f [id=789c86929bf94d2ca3fb5289f598e292]) successfully deleted
W20251026 02:56:27.274933 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 155a23a93358487a9564bb5ac175b4eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 155a23a93358487a9564bb5ac175b4eb already in progress: creating tablet
I20251026 02:56:27.275960 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 81530bbf4a2d4e908add683475e594fb
I20251026 02:56:27.276613 1661 catalog_manager.cc:6719] ReplaceTablet: tablet 81530bbf4a2d4e908add683475e594fb deleted and replaced by tablet 95c9fbae62db4850b0f02a2443a3432b
I20251026 02:56:27.276830 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 81530bbf4a2d4e908add683475e594fb with delete_type TABLET_DATA_DELETED (replaced by tablet 95c9fbae62db4850b0f02a2443a3432b) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.280652 2376 tablet_replica.cc:333] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:27.280906 2376 raft_consensus.cc:2243] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:27.281082 2376 raft_consensus.cc:2272] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:27.282418 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 882fe81da21344ed99ea735fa79f73c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 882fe81da21344ed99ea735fa79f73c0 already in progress: creating tablet
I20251026 02:56:27.282780 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.285167 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e56b259a314422aba35eb129021bbeb) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.287531 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6ce7e6a26747479bb5b4a5b04d37105a" } new_table_name: "default.Table_af9006c41e484127a39c22ff0130d281" modify_external_catalogs: true
I20251026 02:56:27.289300 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_840a6bd369ab416191c1f0ee15bd50ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.289980 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_840a6bd369ab416191c1f0ee15bd50ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.290534 2378 tablet_bootstrap.cc:492] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:27.290666 2378 ts_tablet_manager.cc:1403] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.143s user 0.006s sys 0.000s
I20251026 02:56:27.291522 2378 raft_consensus.cc:359] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.291712 2378 raft_consensus.cc:385] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.291808 2378 raft_consensus.cc:740] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.292186 2378 consensus_queue.cc:260] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.292742 2378 ts_tablet_manager.cc:1434] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:27.293066 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e99391c4c0f74eb3ac3b62cedada8612"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.293748 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e99391c4c0f74eb3ac3b62cedada8612 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.293910 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5f7b25fab37c4983acf922bd78374da6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.294399 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f7b25fab37c4983acf922bd78374da6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.295432 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.295965 1661 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 1d8cdf79a97149618bcaf0fefee1dfeb (table default.Table_424693529d0d4c829755adc2c9835c0e [id=690ec15818d34cdfa990c8fcd36a19cd]). Expected version 1 got 0
I20251026 02:56:27.296509 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_78eb65a5fd1e42d5bdfec024ecd4e1ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.296975 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78eb65a5fd1e42d5bdfec024ecd4e1ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.297375 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3dd70def7f1046ea9f2b5bbb7a74e4cb" } modify_external_catalogs: true
I20251026 02:56:27.297564 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3dd70def7f1046ea9f2b5bbb7a74e4cb" } modify_external_catalogs: true
I20251026 02:56:27.297987 2165 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.299391 2376 ts_tablet_manager.cc:1916] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:27.301542 2194 leader_election.cc:343] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.302299 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8667ffa25f2b478a987826d486400bbe" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:27.302440 2194 leader_election.cc:343] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:27.302812 2194 leader_election.cc:343] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.303709 2631 leader_election.cc:290] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:27.307310 1916 leader_election.cc:343] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 8667ffa25f2b478a987826d486400bbe
I20251026 02:56:27.309051 2372 tablet_replica.cc:333] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:27.310622 2378 tablet_bootstrap.cc:492] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:27.310616 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "740f7c7f186f45ceaa4c40e76419e3a1" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:27.313066 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet ced7ad6a601d432cbc13b3199a0506e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.316856 1915 leader_election.cc:304] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:27.317365 2661 raft_consensus.cc:2749] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:27.313582 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 7845ebd3e9ca4897b8fb19a48a010315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.316079 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 155a23a93358487a9564bb5ac175b4eb with delete_type TABLET_DATA_DELETED (replaced by tablet 386e1bd4635b4a7e8e7f249b577fd978) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.317976 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 81530bbf4a2d4e908add683475e594fb with delete_type TABLET_DATA_DELETED (replaced by tablet 95c9fbae62db4850b0f02a2443a3432b) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.318126 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet ced7ad6a601d432cbc13b3199a0506e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.318392 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.318970 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dec83bfb9e946db90c10d526af2dfca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.319419 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee with delete_type TABLET_DATA_DELETED (replaced by tablet 634d913c98c842a49d97a54f23752d26) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.319621 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.326020 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 47037db3602246a2ab6224557286162f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.326139 2378 tablet_bootstrap.cc:654] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:27.327441 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 81530bbf4a2d4e908add683475e594fb with delete_type TABLET_DATA_DELETED (replaced by tablet 95c9fbae62db4850b0f02a2443a3432b) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.329432 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet c570e5f9a93c45bda32ec92c68794714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.331631 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 8dbc12d49b094dbeadad2e76916ba9af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.332577 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_292ad60362294da0976fd8fe6b180c0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.333097 2089 tablet_service.cc:1552] Processing DeleteTablet for tablet c570e5f9a93c45bda32ec92c68794714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:27.333235 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_292ad60362294da0976fd8fe6b180c0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.333534 2372 raft_consensus.cc:2243] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:27.333726 2372 raft_consensus.cc:2272] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:27.334542 2372 ts_tablet_manager.cc:1916] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:27.341042 2378 tablet_bootstrap.cc:492] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:27.342053 2375 tablet_bootstrap.cc:654] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:27.342790 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8dbc12d49b094dbeadad2e76916ba9af" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:27.343554 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a821ed8a686c452a975acb96b5c74778" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:27.343706 2378 ts_tablet_manager.cc:1403] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20251026 02:56:27.343727 2119 raft_consensus.cc:3060] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:27.343966 2120 raft_consensus.cc:2468] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:27.344581 2378 raft_consensus.cc:359] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.344791 2378 raft_consensus.cc:385] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.344875 2378 raft_consensus.cc:740] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.344995 2118 raft_consensus.cc:2468] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:27.345122 2378 consensus_queue.cc:260] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.351473 2378 ts_tablet_manager.cc:1434] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20251026 02:56:27.350749 2053 consensus_peers.cc:597] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:27.355234 2376 ts_tablet_manager.cc:1929] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:27.355408 2376 log.cc:1199] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/20dec74440cf4bb9949c462ec51e9889
I20251026 02:56:27.355773 2376 ts_tablet_manager.cc:1950] T 20dec74440cf4bb9949c462ec51e9889 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:27.356360 2378 tablet_bootstrap.cc:492] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:27.359393 2195 leader_election.cc:304] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:27.360571 2440 raft_consensus.cc:2804] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:27.360865 1979 raft_consensus.cc:3060] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:27.360989 2440 raft_consensus.cc:493] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:27.361105 2440 raft_consensus.cc:3060] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:27.365793 2119 raft_consensus.cc:2468] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:27.365948 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_828c16a96a47489f835a891f34d707b1" } modify_external_catalogs: true
I20251026 02:56:27.366191 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_828c16a96a47489f835a891f34d707b1" } modify_external_catalogs: true
W20251026 02:56:27.366465 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2524ee0c964345459c3e62f1e39b8f9e consensus state has no leader
I20251026 02:56:27.370237 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet ec13c5386abe414ba25589e03d5b850a from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:27.371327 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 86e483c52e514f1da54335479f59d68e (DEFAULT_TABLE table=default.Table_6d91747dd6164f0d84509497a93aeeea [id=f33faf56d22346bdaa270b22e673315a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.371796 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86e483c52e514f1da54335479f59d68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.373004 1660 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 1d8cdf79a97149618bcaf0fefee1dfeb (table default.Table_424693529d0d4c829755adc2c9835c0e [id=690ec15818d34cdfa990c8fcd36a19cd]). Expected version 1 got 0
I20251026 02:56:27.373282 1660 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet a821ed8a686c452a975acb96b5c74778 (table default.Table_00dc4c49a19246889353df2157eebf15 [id=347d712aca4e477c89805846800fb824]). Expected version 1 got 0
I20251026 02:56:27.373684 1660 catalog_manager.cc:5649] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:27.374120 1660 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 340bf8c78a88413299b61a52f0a578f3 (table default.Table_1361fbf89aa04321a7876c1e43e27089 [id=c162ef89d2f14e91a24a6d8c2c37c261]). Expected version 4 got 0
I20251026 02:56:27.374538 2089 tablet_service.cc:1505] Processing CreateTablet for tablet 8c1e63ac8c5645f3ba1578792c0a0dae (DEFAULT_TABLE table=default.Table_528af198a403496a8c1b27fa5821123b [id=e6aed02773494e3c8694deedf311adc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.374977 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c1e63ac8c5645f3ba1578792c0a0dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.376332 2824 raft_consensus.cc:493] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:27.376509 2824 raft_consensus.cc:515] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.376714 2087 tablet_service.cc:1505] Processing CreateTablet for tablet f719c720cc474181a63786a3e248ab2c (DEFAULT_TABLE table=default.Table_8b019eeaa6074ffb87c0935572a36ed7 [id=67731996dfc04572b0dfebf0edb71942]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.377110 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f719c720cc474181a63786a3e248ab2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.377254 2824 leader_election.cc:290] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:27.380712 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 2524ee0c964345459c3e62f1e39b8f9e
I20251026 02:56:27.381239 1657 catalog_manager.cc:6719] ReplaceTablet: tablet 2524ee0c964345459c3e62f1e39b8f9e deleted and replaced by tablet 5a00bc04b3654f90af342986751344e6
I20251026 02:56:27.381732 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_78eb65a5fd1e42d5bdfec024ecd4e1ba" } new_table_name: "default.Table_26257eca63ce4baaa75beae2e9db1340" modify_external_catalogs: true
I20251026 02:56:27.384156 1955 tablet_service.cc:1505] Processing CreateTablet for tablet f719c720cc474181a63786a3e248ab2c (DEFAULT_TABLE table=default.Table_8b019eeaa6074ffb87c0935572a36ed7 [id=67731996dfc04572b0dfebf0edb71942]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.384370 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 2524ee0c964345459c3e62f1e39b8f9e with delete_type TABLET_DATA_DELETED (replaced by tablet 5a00bc04b3654f90af342986751344e6) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.384619 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f719c720cc474181a63786a3e248ab2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.384667 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 34c6a7c774044411a4733b115080c4c6 (DEFAULT_TABLE table=default.Table_f439acecb4bb4404a14e11f5824c654b [id=d6f9d1f56c154ea3a78a6677347c4a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.384987 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34c6a7c774044411a4733b115080c4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.386785 2378 tablet_bootstrap.cc:654] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:27.388542 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 8c1e63ac8c5645f3ba1578792c0a0dae (DEFAULT_TABLE table=default.Table_528af198a403496a8c1b27fa5821123b [id=e6aed02773494e3c8694deedf311adc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.389379 1661 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]). Expected version 1 got 0
I20251026 02:56:27.388906 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c1e63ac8c5645f3ba1578792c0a0dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.389528 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (DEFAULT_TABLE table=default.Table_b4f29de6c177447f81779bfa9e30ac2d [id=b5d91d3e8a2643c09fee85976972391a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.392664 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7abcdac4a70b48ad9fbba1bc8a60283b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.393360 1661 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]). Expected version 1 got 0
I20251026 02:56:27.396750 2376 tablet_replica.cc:333] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:27.396950 2376 raft_consensus.cc:2243] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:27.397154 2376 raft_consensus.cc:2272] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:27.398355 2378 tablet_bootstrap.cc:492] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:27.398488 2378 ts_tablet_manager.cc:1403] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20251026 02:56:27.399330 2631 consensus_queue.cc:1048] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.035s
I20251026 02:56:27.400802 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 4db275efdd344817b11a2168675cf95e (DEFAULT_TABLE table=default.Table_8a66b2dca58445b1877938b41d7b4507 [id=1fd74186ea10498599435dccd7330d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.401250 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db275efdd344817b11a2168675cf95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.404402 2440 raft_consensus.cc:515] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.404959 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d4304259ac684b8097cdbf5511ab67d7" } modify_external_catalogs: true
I20251026 02:56:27.405081 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_72d8711b32e2406ca3f8f4672cb31ba7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.405200 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d4304259ac684b8097cdbf5511ab67d7" } modify_external_catalogs: true
I20251026 02:56:27.405444 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_672ee2e3775346679182f116bb01cece" } new_table_name: "default.Table_8a002df959a4429eb9570d198b363d39" modify_external_catalogs: true
W20251026 02:56:27.405658 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72d8711b32e2406ca3f8f4672cb31ba7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.404981 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b0a34356c6e54ae79696bc5edfeb8aa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.406818 2440 leader_election.cc:290] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:27.406754 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_000a2b7754d04bd197fca93eab72c154"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.407253 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0a34356c6e54ae79696bc5edfeb8aa1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:27.407514 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_000a2b7754d04bd197fca93eab72c154 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.409382 2376 ts_tablet_manager.cc:1916] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:27.409879 1950 tablet_service.cc:1505] Processing CreateTablet for tablet a012d868a78f4324a92db83c81931b9d (DEFAULT_TABLE table=default.Table_3dd70def7f1046ea9f2b5bbb7a74e4cb [id=b4e3ea5575a140ac85b4c0cbd57c9ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.410133 2381 tablet_bootstrap.cc:492] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:27.410319 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a012d868a78f4324a92db83c81931b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.410306 2381 ts_tablet_manager.cc:1403] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.277s user 0.011s sys 0.005s
I20251026 02:56:27.414093 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 48105e54d6324622a032e1288a46188c (DEFAULT_TABLE table=default.Table_3bc5fdbe07c348cf8f30654554a8b95a [id=4538b44c47204ba1bcd163a3eba3d951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.414589 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48105e54d6324622a032e1288a46188c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.414649 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 386e1bd4635b4a7e8e7f249b577fd978 (DEFAULT_TABLE table=default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.415047 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 386e1bd4635b4a7e8e7f249b577fd978. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.421155 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet 0020cb1f190c49978c5322d387a47f71 with delete_type TABLET_DATA_DELETED (replaced by tablet 86d8e1e21c5f440d8baf76c357d01ad2) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.421487 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7845ebd3e9ca4897b8fb19a48a010315" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:27.421785 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8dbc12d49b094dbeadad2e76916ba9af" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:27.425282 2053 leader_election.cc:343] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:27.425784 2194 leader_election.cc:343] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.425760 2377 ts_tablet_manager.cc:1929] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:27.414165 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 8667ffa25f2b478a987826d486400bbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:27.429176 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet d65b0d79e2e54af9810022b683c3288b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d65b0d79e2e54af9810022b683c3288b already in progress: creating tablet
W20251026 02:56:27.429517 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet f85c2745a371455a8dda67f31a19c9d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f85c2745a371455a8dda67f31a19c9d5 already in progress: creating tablet
W20251026 02:56:27.429731 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet e62064fbdffb4954a3a1227efd6327ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e62064fbdffb4954a3a1227efd6327ee already in progress: creating tablet
W20251026 02:56:27.429968 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ca4611c0a494425ca50d599b310872c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca4611c0a494425ca50d599b310872c8 already in progress: creating tablet
W20251026 02:56:27.430176 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ae95654d129441e08fcb3f3b355ffc22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae95654d129441e08fcb3f3b355ffc22 already in progress: creating tablet
W20251026 02:56:27.430405 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 5caeb11322fe48348ebb523967152187 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5caeb11322fe48348ebb523967152187 already in progress: creating tablet
W20251026 02:56:27.430581 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 47037db3602246a2ab6224557286162f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47037db3602246a2ab6224557286162f already in progress: creating tablet
W20251026 02:56:27.430757 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 6a3491b303f94fe2a2c838297f73d464 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a3491b303f94fe2a2c838297f73d464 already in progress: creating tablet
I20251026 02:56:27.430384 2381 raft_consensus.cc:359] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.431856 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 86e483c52e514f1da54335479f59d68e (DEFAULT_TABLE table=default.Table_6d91747dd6164f0d84509497a93aeeea [id=f33faf56d22346bdaa270b22e673315a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.431977 2381 raft_consensus.cc:385] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.432315 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86e483c52e514f1da54335479f59d68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.435490 2381 raft_consensus.cc:740] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.435850 2381 consensus_queue.cc:260] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.420405 2195 leader_election.cc:304] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:27.436489 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ec13c5386abe414ba25589e03d5b850a
I20251026 02:56:27.437052 1662 catalog_manager.cc:6719] ReplaceTablet: tablet ec13c5386abe414ba25589e03d5b850a deleted and replaced by tablet 6699b50d87964f6a89c365171253ad25
I20251026 02:56:27.437494 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet ac19c5a847454a23ab16831198ff2a20 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:27.438203 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5ad87a5fcdd745529b77d2ffd8c2a4ff" } new_table_name: "default.Table_f51a5406d89743649dab37ef175a4666" modify_external_catalogs: true
I20251026 02:56:27.438397 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 741792cf96164695b03de3fb86b12602
I20251026 02:56:27.439217 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_feb822201306444ebe478734d286a667"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.439771 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_feb822201306444ebe478734d286a667 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.440742 2308 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.440778 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6fd0ab283f2c4b50a4e6cc4cb073c742" } new_table_name: "default.Table_8ffce6cf6b2d44aa8a9459e834de1d80" modify_external_catalogs: true
W20251026 02:56:27.442665 1647 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet e701d4023fee45d18c48997aab3227c2 (table default.Table_f83c29969baa4d62bca3154cc3d2cc33 [id=de836edc225f41d8b0c68748936306ad], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e701d4023fee45d18c48997aab3227c2 consensus state has no leader
I20251026 02:56:27.442987 2631 raft_consensus.cc:2804] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:27.443106 2631 raft_consensus.cc:493] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:27.443255 2631 raft_consensus.cc:3060] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:27.445055 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f4fc99282b5643e89641bb7082d165a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.445641 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4fc99282b5643e89641bb7082d165a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.446537 2631 raft_consensus.cc:515] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.447178 2631 leader_election.cc:290] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:27.447750 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 48105e54d6324622a032e1288a46188c (DEFAULT_TABLE table=default.Table_3bc5fdbe07c348cf8f30654554a8b95a [id=4538b44c47204ba1bcd163a3eba3d951]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:27.447921 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 87816.9us
I20251026 02:56:27.448307 2195 leader_election.cc:304] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:27.448355 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48105e54d6324622a032e1288a46188c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.448691 2440 consensus_queue.cc:1048] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:27.449105 2440 raft_consensus.cc:2804] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:27.449252 2440 raft_consensus.cc:697] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:27.449494 2440 consensus_queue.cc:237] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.452595 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 2524ee0c964345459c3e62f1e39b8f9e with delete_type TABLET_DATA_DELETED (replaced by tablet 5a00bc04b3654f90af342986751344e6) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.453089 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 386e1bd4635b4a7e8e7f249b577fd978 (DEFAULT_TABLE table=default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.453475 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 386e1bd4635b4a7e8e7f249b577fd978. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:27.454327 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fb2ff87cf0248a0a4a77f2733ef867b has no consensus state
W20251026 02:56:27.456496 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de8799a5af2b42cc92ccc0195cff98e7 consensus state has no leader
I20251026 02:56:27.456537 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet ec13c5386abe414ba25589e03d5b850a with delete_type TABLET_DATA_DELETED (replaced by tablet 6699b50d87964f6a89c365171253ad25) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.456692 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 86e483c52e514f1da54335479f59d68e (DEFAULT_TABLE table=default.Table_6d91747dd6164f0d84509497a93aeeea [id=f33faf56d22346bdaa270b22e673315a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.456946 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 741792cf96164695b03de3fb86b12602 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.457095 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86e483c52e514f1da54335479f59d68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.457358 2085 tablet_service.cc:1505] Processing CreateTablet for tablet a012d868a78f4324a92db83c81931b9d (DEFAULT_TABLE table=default.Table_3dd70def7f1046ea9f2b5bbb7a74e4cb [id=b4e3ea5575a140ac85b4c0cbd57c9ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.457718 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a012d868a78f4324a92db83c81931b9d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:27.460332 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dba102838945453cbf44e1cfe505a52c consensus state has no leader
I20251026 02:56:27.462226 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a6f9ae327aa4401baf66a737eaa93cf6
I20251026 02:56:27.463091 2115 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "740f7c7f186f45ceaa4c40e76419e3a1" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:27.463348 2115 raft_consensus.cc:3060] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:27.463608 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a012d868a78f4324a92db83c81931b9d
I20251026 02:56:27.464563 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8dbc12d49b094dbeadad2e76916ba9af" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:27.464792 2118 raft_consensus.cc:3060] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:27.465869 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bfa68f2c942c43e2a02e9d333bdfa2a2 consensus state has no leader
I20251026 02:56:27.466269 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 34c6a7c774044411a4733b115080c4c6 (DEFAULT_TABLE table=default.Table_f439acecb4bb4404a14e11f5824c654b [id=d6f9d1f56c154ea3a78a6677347c4a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.466742 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34c6a7c774044411a4733b115080c4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.468828 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 23062e7292f44ad5ad9dfc22a401741c (DEFAULT_TABLE table=default.Table_dc751f63612b4379a8d8069bc750d1f4 [id=2e17bae10e0b48a492f5ee0bed7d00d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.436719 2381 ts_tablet_manager.cc:1434] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20251026 02:56:27.469450 2381 tablet_bootstrap.cc:492] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:27.470144 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.471587 2381 tablet_bootstrap.cc:654] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:27.472514 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (DEFAULT_TABLE table=default.Table_b4f29de6c177447f81779bfa9e30ac2d [id=b5d91d3e8a2643c09fee85976972391a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.473244 2232 tablet_service.cc:1505] Processing CreateTablet for tablet f719c720cc474181a63786a3e248ab2c (DEFAULT_TABLE table=default.Table_8b019eeaa6074ffb87c0935572a36ed7 [id=67731996dfc04572b0dfebf0edb71942]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.472864 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23062e7292f44ad5ad9dfc22a401741c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.473621 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f719c720cc474181a63786a3e248ab2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.475247 2824 raft_consensus.cc:493] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:27.475436 2824 raft_consensus.cc:515] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.476092 2824 leader_election.cc:290] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:27.477447 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7abcdac4a70b48ad9fbba1bc8a60283b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.480415 2631 raft_consensus.cc:493] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:27.480609 2631 raft_consensus.cc:515] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.481455 2631 leader_election.cc:290] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:27.481909 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4011930a277d410f863a1558416c4b0f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:27.483688 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 4db275efdd344817b11a2168675cf95e (DEFAULT_TABLE table=default.Table_8a66b2dca58445b1877938b41d7b4507 [id=1fd74186ea10498599435dccd7330d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.484092 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db275efdd344817b11a2168675cf95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.489634 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet ec13c5386abe414ba25589e03d5b850a with delete_type TABLET_DATA_DELETED (replaced by tablet 6699b50d87964f6a89c365171253ad25) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.490497 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet a6f9ae327aa4401baf66a737eaa93cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.490933 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet a012d868a78f4324a92db83c81931b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.492964 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (DEFAULT_TABLE table=default.Table_b4f29de6c177447f81779bfa9e30ac2d [id=b5d91d3e8a2643c09fee85976972391a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.493393 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7abcdac4a70b48ad9fbba1bc8a60283b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.493717 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 2524ee0c964345459c3e62f1e39b8f9e with delete_type TABLET_DATA_DELETED (replaced by tablet 5a00bc04b3654f90af342986751344e6) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.494279 2372 ts_tablet_manager.cc:1929] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:27.494439 2372 log.cc:1199] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/9e5bb0a6a9f947cf9ce119c1fc91fbf5
I20251026 02:56:27.494446 2236 tablet_service.cc:1505] Processing CreateTablet for tablet a012d868a78f4324a92db83c81931b9d (DEFAULT_TABLE table=default.Table_3dd70def7f1046ea9f2b5bbb7a74e4cb [id=b4e3ea5575a140ac85b4c0cbd57c9ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.494772 2372 ts_tablet_manager.cc:1950] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:27.494833 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a012d868a78f4324a92db83c81931b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.493307 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 34c6a7c774044411a4733b115080c4c6 (DEFAULT_TABLE table=default.Table_f439acecb4bb4404a14e11f5824c654b [id=d6f9d1f56c154ea3a78a6677347c4a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.496591 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34c6a7c774044411a4733b115080c4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.498104 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet a012d868a78f4324a92db83c81931b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.498720 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 23062e7292f44ad5ad9dfc22a401741c (DEFAULT_TABLE table=default.Table_dc751f63612b4379a8d8069bc750d1f4 [id=2e17bae10e0b48a492f5ee0bed7d00d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.499084 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23062e7292f44ad5ad9dfc22a401741c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.500922 2115 raft_consensus.cc:2468] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:27.495059 2308 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.506299 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 48105e54d6324622a032e1288a46188c (DEFAULT_TABLE table=default.Table_3bc5fdbe07c348cf8f30654554a8b95a [id=4538b44c47204ba1bcd163a3eba3d951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.506757 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48105e54d6324622a032e1288a46188c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.504153 2118 raft_consensus.cc:2468] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:27.512269 2378 raft_consensus.cc:359] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.512563 2378 raft_consensus.cc:385] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.512626 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "740f7c7f186f45ceaa4c40e76419e3a1" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:27.516906 1979 raft_consensus.cc:3060] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:27.517359 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 8c1e63ac8c5645f3ba1578792c0a0dae (DEFAULT_TABLE table=default.Table_528af198a403496a8c1b27fa5821123b [id=e6aed02773494e3c8694deedf311adc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.517889 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c1e63ac8c5645f3ba1578792c0a0dae. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:27.520419 2053 leader_election.cc:343] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.520762 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7845ebd3e9ca4897b8fb19a48a010315" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:27.526173 2377 log.cc:1199] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/cfed3523205548508d1101586ee3e2a9
I20251026 02:56:27.526703 2377 ts_tablet_manager.cc:1950] T cfed3523205548508d1101586ee3e2a9 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:27.527138 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 290) took 1406 ms. Trace:
I20251026 02:56:27.527236 2377 rpcz_store.cc:276] 1026 02:56:26.120744 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.120802 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:27.192953 (+1072151us) ts_tablet_manager.cc:1151] Deleting tablet cfed3523205548508d1101586ee3e2a9
1026 02:56:27.192968 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:27.357288 (+164320us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:27.425617 (+ 68329us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:27.527079 (+101462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1071972}
I20251026 02:56:27.527745 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 287) took 1408 ms. Trace:
I20251026 02:56:27.527843 2377 rpcz_store.cc:276] 1026 02:56:26.118937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.141444 (+ 22507us) service_pool.cc:225] Handling call
1026 02:56:27.527573 (+1386129us) ts_tablet_manager.cc:1151] Deleting tablet 1d7ffdaeec2240dcbe4cafdb96073a7d
1026 02:56:27.527582 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:27.527690 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1385902}
I20251026 02:56:27.528909 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 1d8cdf79a97149618bcaf0fefee1dfeb (table default.Table_424693529d0d4c829755adc2c9835c0e [id=690ec15818d34cdfa990c8fcd36a19cd]). Expected version 1 got 0
I20251026 02:56:27.529150 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 22e4a433801a4cb58062d3e76c760b69 (table default.Table_f7c216f3ebd34c718ceef042a435b4cd [id=b4c0611288a949378eed16d32b5170b9]). Expected version 2 got 0
I20251026 02:56:27.529191 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet ec13c5386abe414ba25589e03d5b850a with delete_type TABLET_DATA_DELETED (replaced by tablet 6699b50d87964f6a89c365171253ad25) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:27.529466 2194 consensus_peers.cc:597] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2bfbe624075463f8b90ed4ec66b784c. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:27.529464 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 903e65fa8c214dc9a50cb97cab101d22 (table default.Table_862bd13f099645e8aafd987e4e850ae6 [id=65b5770c4b9243b892bfab963a439bb9]). Expected version 3 got 0
I20251026 02:56:27.529680 1656 catalog_manager.cc:5649] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:27.530099 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 9a08e8b7eb1949408f2151f2d7257e9d (table default.Table_5f363b68bc2c4e7685f25e4eefa532a8 [id=43cb7f538f824ae38059645230cc7e94]). Expected version 3 got 0
I20251026 02:56:27.530282 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 741792cf96164695b03de3fb86b12602 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:27.531221 2194 leader_election.cc:343] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.532905 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet a6f9ae327aa4401baf66a737eaa93cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.533603 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_528abc1c5aa443a89795cc31b809be9a" } modify_external_catalogs: true
I20251026 02:56:27.533860 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_528abc1c5aa443a89795cc31b809be9a" } modify_external_catalogs: true
I20251026 02:56:27.534735 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_528af198a403496a8c1b27fa5821123b" } modify_external_catalogs: true
I20251026 02:56:27.534919 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_528af198a403496a8c1b27fa5821123b" } modify_external_catalogs: true
I20251026 02:56:27.536091 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet cfc3aed64ad0439cb22fd6815c973d87 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:27.414651 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 23062e7292f44ad5ad9dfc22a401741c (DEFAULT_TABLE table=default.Table_dc751f63612b4379a8d8069bc750d1f4 [id=2e17bae10e0b48a492f5ee0bed7d00d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.538049 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23062e7292f44ad5ad9dfc22a401741c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.540426 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 741792cf96164695b03de3fb86b12602 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.540421 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 386e1bd4635b4a7e8e7f249b577fd978 (DEFAULT_TABLE table=default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.543335 2376 ts_tablet_manager.cc:1929] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:27.545508 2376 log.cc:1199] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/7176cbe8f64546b7871d047c0e0e5b2a
I20251026 02:56:27.546507 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8c1e63ac8c5645f3ba1578792c0a0dae
I20251026 02:56:27.547395 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c1e63ac8c5645f3ba1578792c0a0dae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.547598 2376 ts_tablet_manager.cc:1950] T 7176cbe8f64546b7871d047c0e0e5b2a P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
W20251026 02:56:27.547595 2194 consensus_peers.cc:597] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:27.548071 2194 leader_election.cc:343] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.548323 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 9156314f98ac4ed9981f6f13fc585043 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.548556 2440 consensus_queue.cc:1048] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:27.549079 2377 tablet_replica.cc:333] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:27.549309 2377 raft_consensus.cc:2243] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:27.549510 2377 raft_consensus.cc:2272] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:27.550135 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 7176cbe8f64546b7871d047c0e0e5b2a (table default.Table_c46d7af7b60a48188efbea88a1b7a72b [id=93bef2a8d7b649e880042bfdbb1e08ac]) successfully deleted
I20251026 02:56:27.550984 2089 tablet_service.cc:1552] Processing DeleteTablet for tablet 8dbc12d49b094dbeadad2e76916ba9af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.551317 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 84ad6c72a8704c148411881760848df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.551422 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet f8533f1c191042ee9f2c0b99972fe931 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.551560 2089 tablet_service.cc:1552] Processing DeleteTablet for tablet 9156314f98ac4ed9981f6f13fc585043 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.551792 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c1e63ac8c5645f3ba1578792c0a0dae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.551999 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 7845ebd3e9ca4897b8fb19a48a010315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:27.552177 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ced7ad6a601d432cbc13b3199a0506e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ced7ad6a601d432cbc13b3199a0506e4 already in progress: creating tablet
I20251026 02:56:27.552809 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet a6f9ae327aa4401baf66a737eaa93cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.553519 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e56b259a314422aba35eb129021bbeb) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.512704 2378 raft_consensus.cc:740] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.554139 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet a012d868a78f4324a92db83c81931b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.552189 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 4db275efdd344817b11a2168675cf95e (DEFAULT_TABLE table=default.Table_8a66b2dca58445b1877938b41d7b4507 [id=1fd74186ea10498599435dccd7330d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.554957 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c1e63ac8c5645f3ba1578792c0a0dae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.557236 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7845ebd3e9ca4897b8fb19a48a010315" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:27.557297 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet cfed3523205548508d1101586ee3e2a9 (table default.Table_e4fbc44faf984982986abfb227d6663d [id=b35a8189bbc04e94bb097c9e13a38ec6]) successfully deleted
W20251026 02:56:27.557593 2195 consensus_peers.cc:597] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2bfbe624075463f8b90ed4ec66b784c. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:27.557583 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d7ffdaeec2240dcbe4cafdb96073a7d already in progress: creating tablet
I20251026 02:56:27.557634 2118 raft_consensus.cc:2468] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:27.558413 2195 leader_election.cc:304] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:27.559026 2631 raft_consensus.cc:2804] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:27.559204 2631 raft_consensus.cc:697] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:27.559652 2377 ts_tablet_manager.cc:1916] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:27.559777 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet cfc3aed64ad0439cb22fd6815c973d87
I20251026 02:56:27.560422 1660 catalog_manager.cc:6719] ReplaceTablet: tablet cfc3aed64ad0439cb22fd6815c973d87 deleted and replaced by tablet 3b77bb6b4722449c9699b548a9ef6314
I20251026 02:56:27.560286 2631 consensus_queue.cc:237] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:27.562201 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet c570e5f9a93c45bda32ec92c68794714 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c570e5f9a93c45bda32ec92c68794714 already in progress: creating tablet
I20251026 02:56:27.562754 2376 tablet_replica.cc:333] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:27.563208 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet cfc3aed64ad0439cb22fd6815c973d87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b77bb6b4722449c9699b548a9ef6314) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.563247 2376 raft_consensus.cc:2243] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:27.563557 2376 raft_consensus.cc:2272] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:27.564678 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet cfc3aed64ad0439cb22fd6815c973d87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b77bb6b4722449c9699b548a9ef6314) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.566877 2378 consensus_queue.cc:260] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.573695 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet d43b6f22dd4f4f9c9657f21514550312
I20251026 02:56:27.569620 2195 leader_election.cc:304] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:27.574441 2440 raft_consensus.cc:2804] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:27.574934 2440 raft_consensus.cc:697] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:27.575413 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet d43b6f22dd4f4f9c9657f21514550312 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.575346 2440 consensus_queue.cc:237] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.578024 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 62d30d79a6e4432a8dbe40a104e87fa0 (DEFAULT_TABLE table=default.Table_5f7b25fab37c4983acf922bd78374da6 [id=777528a5bf944ecb96ebdbea9e7ee559]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.578527 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d30d79a6e4432a8dbe40a104e87fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.580801 2195 leader_election.cc:304] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:27.581146 2759 raft_consensus.cc:2804] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:27.581270 2759 raft_consensus.cc:493] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:27.581367 2759 raft_consensus.cc:3060] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:27.587276 2759 raft_consensus.cc:515] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.588546 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 9c7efb70c0804c149f36440a4a9e6ef8 (DEFAULT_TABLE table=default.Table_e99391c4c0f74eb3ac3b62cedada8612 [id=4397550364664316a6f1e96fb17d9509]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.589039 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c7efb70c0804c149f36440a4a9e6ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.588897 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8d1fb0a5abc3483e85ee11cfc62a0b13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.589119 2759 leader_election.cc:290] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:27.589295 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8fbaf8dac35d47848025cbb47c261c08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.589525 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d1fb0a5abc3483e85ee11cfc62a0b13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:27.590369 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fbaf8dac35d47848025cbb47c261c08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:27.595422 2194 consensus_peers.cc:597] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:27.598016 2375 tablet_bootstrap.cc:492] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:27.598233 2375 ts_tablet_manager.cc:1403] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.445s user 0.004s sys 0.001s
I20251026 02:56:27.599009 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4c889f76507c4e3e975fd19c21a2a37a" } new_table_name: "default.Table_73e25969b3304d3a8d466a21f9445fcb" modify_external_catalogs: true
I20251026 02:56:27.599092 2375 raft_consensus.cc:359] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.599347 2375 raft_consensus.cc:385] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.599428 2375 raft_consensus.cc:740] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.599663 2375 consensus_queue.cc:260] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.600390 2375 ts_tablet_manager.cc:1434] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251026 02:56:27.600492 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f4bddfca37d64369b99a1fdc0102ca35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.600730 2375 tablet_bootstrap.cc:492] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:27.601074 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4bddfca37d64369b99a1fdc0102ca35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.601511 2381 tablet_bootstrap.cc:492] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:27.601704 2381 ts_tablet_manager.cc:1403] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.132s user 0.006s sys 0.000s
I20251026 02:56:27.602501 2375 tablet_bootstrap.cc:654] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:27.602449 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_874fdf4c2b4e4ace9b87c72de1e573a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.603015 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_874fdf4c2b4e4ace9b87c72de1e573a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.602890 2381 raft_consensus.cc:359] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.604765 2372 tablet_replica.cc:333] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:27.605049 2372 raft_consensus.cc:2243] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:27.605355 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 (table default.Table_25607539f64440cb8ff2c24a0a3a3bdd [id=5dbfb5d912ba433baa46bd52bbf6dbee]) successfully deleted
I20251026 02:56:27.585420 2027 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:27.570655 2378 ts_tablet_manager.cc:1434] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.172s user 0.001s sys 0.002s
I20251026 02:56:27.610400 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5a3412a706124204a0379462d7628a17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.611007 2378 tablet_bootstrap.cc:492] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:27.611047 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a3412a706124204a0379462d7628a17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.587011 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet d43b6f22dd4f4f9c9657f21514550312 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.611749 2381 raft_consensus.cc:385] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.611881 2381 raft_consensus.cc:740] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.595238 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 62d30d79a6e4432a8dbe40a104e87fa0 (DEFAULT_TABLE table=default.Table_5f7b25fab37c4983acf922bd78374da6 [id=777528a5bf944ecb96ebdbea9e7ee559]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.612356 2381 consensus_queue.cc:260] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.613157 2381 ts_tablet_manager.cc:1434] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20251026 02:56:27.596938 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db275efdd344817b11a2168675cf95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.619860 2263 raft_consensus.cc:1275] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Refusing update from remote peer 0b6b0a6c4e144982b67980118cddba48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:27.620153 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet cfc3aed64ad0439cb22fd6815c973d87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b77bb6b4722449c9699b548a9ef6314) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.620265 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4011930a277d410f863a1558416c4b0f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:27.620432 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet d43b6f22dd4f4f9c9657f21514550312 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.620581 2261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7845ebd3e9ca4897b8fb19a48a010315" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:27.611563 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7845ebd3e9ca4897b8fb19a48a010315" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:27.620803 2261 raft_consensus.cc:2393] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b6b0a6c4e144982b67980118cddba48 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:27.620787 2118 raft_consensus.cc:3060] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:27.620900 2260 raft_consensus.cc:1275] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Refusing update from remote peer 0b6b0a6c4e144982b67980118cddba48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:27.621898 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 386e1bd4635b4a7e8e7f249b577fd978. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.623736 2118 raft_consensus.cc:2468] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:27.624130 2372 raft_consensus.cc:2272] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:27.624751 2372 ts_tablet_manager.cc:1916] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:27.629359 1663 catalog_manager.cc:5649] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:27.629757 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet a821ed8a686c452a975acb96b5c74778 (table default.Table_00dc4c49a19246889353df2157eebf15 [id=347d712aca4e477c89805846800fb824]). Expected version 1 got 0
W20251026 02:56:27.630885 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 049a915b35224cdaa38df77c734a0f01 consensus state has no leader
I20251026 02:56:27.631596 2378 tablet_bootstrap.cc:654] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:27.635063 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d30d79a6e4432a8dbe40a104e87fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.637552 2375 tablet_bootstrap.cc:492] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:27.637625 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 3c3050e72d42417ab58a05f7e3aa56b3 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:27.637827 2375 ts_tablet_manager.cc:1403] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.000s
I20251026 02:56:27.638900 2375 raft_consensus.cc:359] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.639114 2375 raft_consensus.cc:385] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.639221 2375 raft_consensus.cc:740] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.639441 2375 consensus_queue.cc:260] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:27.642850 2053 consensus_peers.cc:597] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:27.649303 2375 ts_tablet_manager.cc:1434] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251026 02:56:27.649665 2375 tablet_bootstrap.cc:492] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:27.651487 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 62d30d79a6e4432a8dbe40a104e87fa0 (DEFAULT_TABLE table=default.Table_5f7b25fab37c4983acf922bd78374da6 [id=777528a5bf944ecb96ebdbea9e7ee559]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.651592 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 9c7efb70c0804c149f36440a4a9e6ef8 (DEFAULT_TABLE table=default.Table_e99391c4c0f74eb3ac3b62cedada8612 [id=4397550364664316a6f1e96fb17d9509]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.651937 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d30d79a6e4432a8dbe40a104e87fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.652944 2375 tablet_bootstrap.cc:654] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:27.653340 2824 consensus_queue.cc:1048] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:27.653203 1663 catalog_manager.cc:5649] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
W20251026 02:56:27.653592 2053 leader_election.cc:343] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.653625 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (table default.Table_fa634331e4bd4c14a102908bc41edba1 [id=1d09f5b9fa164ac783f0be9332eac4c3]). Expected version 4 got 0
I20251026 02:56:27.653753 2053 leader_election.cc:304] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:27.653997 2824 raft_consensus.cc:2749] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:27.655226 2053 leader_election.cc:304] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:27.655515 2824 raft_consensus.cc:2749] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:27.656939 2824 consensus_queue.cc:1048] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:27.658243 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 1d8cdf79a97149618bcaf0fefee1dfeb (table default.Table_424693529d0d4c829755adc2c9835c0e [id=690ec15818d34cdfa990c8fcd36a19cd]). Expected version 1 got 0
I20251026 02:56:27.658892 2195 leader_election.cc:304] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:27.660602 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1fba8934d98d4fdb8b2c9a0b5d01e810" } modify_external_catalogs: true
I20251026 02:56:27.660838 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1fba8934d98d4fdb8b2c9a0b5d01e810" } modify_external_catalogs: true
I20251026 02:56:27.661798 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4f8511a57eb54e5eaf42a6361676ff21" } new_table_name: "default.Table_653a3f85c24140cdb62a3e9f3ea95fd4" modify_external_catalogs: true
I20251026 02:56:27.663597 2165 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.664974 2879 raft_consensus.cc:2804] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:27.665221 2879 raft_consensus.cc:697] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:27.665768 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1bc1610289a448d99b780c0e289b7d58" } new_table_name: "default.Table_dafd0323dc7542da8dca42d770f1455f" modify_external_catalogs: true
I20251026 02:56:27.666064 2879 consensus_queue.cc:237] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.666777 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0dfae834c6294764ae2c58c6a11c5280"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.667091 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet aa399e6407fd4c4d9871d3d5ccf894ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:27.667586 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dfae834c6294764ae2c58c6a11c5280 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.668159 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_87f51d8912864f0aa7ef8b490392611e" } new_table_name: "default.Table_7c080afce3474539b73dd8c734adef21" modify_external_catalogs: true
I20251026 02:56:27.668874 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_559b75aad2cb44b196cc0ab1dbf2c550"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.669541 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a72502f837fb4542b1f00bab5da37724" } modify_external_catalogs: true
I20251026 02:56:27.669744 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a72502f837fb4542b1f00bab5da37724" } modify_external_catalogs: true
I20251026 02:56:27.670603 2381 tablet_bootstrap.cc:492] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:27.671005 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b555f404492a47808d8a9a168a45ef2f" } modify_external_catalogs: true
I20251026 02:56:27.671169 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b555f404492a47808d8a9a168a45ef2f" } modify_external_catalogs: true
I20251026 02:56:27.672394 2381 tablet_bootstrap.cc:654] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:27.675009 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3c3050e72d42417ab58a05f7e3aa56b3
I20251026 02:56:27.675495 1658 catalog_manager.cc:6719] ReplaceTablet: tablet 3c3050e72d42417ab58a05f7e3aa56b3 deleted and replaced by tablet 9d31527b9ceb4b17a55fd93aee79b9cd
I20251026 02:56:27.677268 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3050e72d42417ab58a05f7e3aa56b3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d31527b9ceb4b17a55fd93aee79b9cd) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.677703 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 9c7efb70c0804c149f36440a4a9e6ef8 (DEFAULT_TABLE table=default.Table_e99391c4c0f74eb3ac3b62cedada8612 [id=4397550364664316a6f1e96fb17d9509]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.678249 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c7efb70c0804c149f36440a4a9e6ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.679102 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7845ebd3e9ca4897b8fb19a48a010315" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:27.679726 2194 leader_election.cc:343] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.680320 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.680505 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5e91141bf6d441ab9d1842e8a859869a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.681120 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e91141bf6d441ab9d1842e8a859869a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:27.682169 2195 consensus_peers.cc:597] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8667ffa25f2b478a987826d486400bbe. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:27.685511 2376 ts_tablet_manager.cc:1916] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:27.693794 2440 consensus_queue.cc:1048] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:27.694852 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3050e72d42417ab58a05f7e3aa56b3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d31527b9ceb4b17a55fd93aee79b9cd) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.698509 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c7efb70c0804c149f36440a4a9e6ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.709414 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 0cf76ef8f6af445eb0ce15c8d79136c5 (DEFAULT_TABLE table=default.Table_672f95f49551403a99355e82b17eaece [id=33334e177bb94b3e9de2a7e7ee59979b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.710122 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cf76ef8f6af445eb0ce15c8d79136c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.719343 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 0cf76ef8f6af445eb0ce15c8d79136c5 (DEFAULT_TABLE table=default.Table_672f95f49551403a99355e82b17eaece [id=33334e177bb94b3e9de2a7e7ee59979b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.719878 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cf76ef8f6af445eb0ce15c8d79136c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.722142 2087 tablet_service.cc:1505] Processing CreateTablet for tablet f5228f482c0b4fcbbdf8ebc31bb62819 (DEFAULT_TABLE table=default.Table_0f53c2a8d63c47769dca3283cb871b9c [id=6300327f9d41469e9e70af59b7789ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.722642 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5228f482c0b4fcbbdf8ebc31bb62819. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.726563 1960 tablet_service.cc:1505] Processing CreateTablet for tablet f5228f482c0b4fcbbdf8ebc31bb62819 (DEFAULT_TABLE table=default.Table_0f53c2a8d63c47769dca3283cb871b9c [id=6300327f9d41469e9e70af59b7789ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.726989 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5228f482c0b4fcbbdf8ebc31bb62819. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:27.669553 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_559b75aad2cb44b196cc0ab1dbf2c550 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.733765 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 99d318aa0d1847c58167bf0af9e75aa2 (DEFAULT_TABLE table=default.Table_62fd5a902d0648cea484f259d02de5f9 [id=3dc4d9941b7147adb84d1754e2783aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.734220 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99d318aa0d1847c58167bf0af9e75aa2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:27.736513 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 21b936fa4f6a4ff6af321971a30cccbe consensus state has no leader
W20251026 02:56:27.738503 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c6e3e47b3e7421e945cfbe671b06abe consensus state has no leader
I20251026 02:56:27.739557 1662 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]). Expected version 1 got 0
W20251026 02:56:27.740530 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce1ab21ddf8e45bc9063b63db9c1b97f consensus state has no leader
I20251026 02:56:27.740882 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_91f57fb05381498c89f99d19bdc7cd9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.741466 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91f57fb05381498c89f99d19bdc7cd9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.743198 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 78a624d839934d3089b64e397d4360af
I20251026 02:56:27.743844 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_461dd114dc774ff7bb820f85a2acc574"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.744406 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_461dd114dc774ff7bb820f85a2acc574 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.744714 2308 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.746757 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4e5fa0edbee549878f707b605c7c0180" } new_table_name: "default.Table_8602d75889ea4576ae575b55fe95e940" modify_external_catalogs: true
I20251026 02:56:27.748337 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 78a624d839934d3089b64e397d4360af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.752130 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet f437d8b5f5784e8fbec10d0700e90348
I20251026 02:56:27.753248 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_72d8711b32e2406ca3f8f4672cb31ba7" } modify_external_catalogs: true
I20251026 02:56:27.753473 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_72d8711b32e2406ca3f8f4672cb31ba7" } modify_external_catalogs: true
I20251026 02:56:27.754285 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 99d318aa0d1847c58167bf0af9e75aa2 (DEFAULT_TABLE table=default.Table_62fd5a902d0648cea484f259d02de5f9 [id=3dc4d9941b7147adb84d1754e2783aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.754745 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99d318aa0d1847c58167bf0af9e75aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.755649 2381 tablet_bootstrap.cc:492] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:27.755798 2381 ts_tablet_manager.cc:1403] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.002s
I20251026 02:56:27.756605 2381 raft_consensus.cc:359] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.756856 2381 raft_consensus.cc:385] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.756968 2381 raft_consensus.cc:740] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.757035 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet f437d8b5f5784e8fbec10d0700e90348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.757273 2381 consensus_queue.cc:260] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.757980 2381 ts_tablet_manager.cc:1434] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:27.758297 2381 tablet_bootstrap.cc:492] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:27.759105 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 78a624d839934d3089b64e397d4360af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.759471 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet f437d8b5f5784e8fbec10d0700e90348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.761587 2376 ts_tablet_manager.cc:1929] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:27.761771 2376 log.cc:1199] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/81530bbf4a2d4e908add683475e594fb
I20251026 02:56:27.762282 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3050e72d42417ab58a05f7e3aa56b3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d31527b9ceb4b17a55fd93aee79b9cd) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.762410 2376 ts_tablet_manager.cc:1950] T 81530bbf4a2d4e908add683475e594fb P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:27.764328 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 9a345fa5edbd48d2891b8b40b09cb3f8 (DEFAULT_TABLE table=default.Table_3d25c519f16647ca962f3862c6cd4f98 [id=eca037eee4b54221b7c9771c390055c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.765241 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 81530bbf4a2d4e908add683475e594fb (table default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]) successfully deleted
I20251026 02:56:27.765290 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet f8533f1c191042ee9f2c0b99972fe931 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.765677 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 78a624d839934d3089b64e397d4360af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.765692 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 9a345fa5edbd48d2891b8b40b09cb3f8 (DEFAULT_TABLE table=default.Table_3d25c519f16647ca962f3862c6cd4f98 [id=eca037eee4b54221b7c9771c390055c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.766191 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a345fa5edbd48d2891b8b40b09cb3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.766217 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 99d318aa0d1847c58167bf0af9e75aa2 (DEFAULT_TABLE table=default.Table_62fd5a902d0648cea484f259d02de5f9 [id=3dc4d9941b7147adb84d1754e2783aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.766651 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99d318aa0d1847c58167bf0af9e75aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.767947 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a345fa5edbd48d2891b8b40b09cb3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.769214 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]). Expected version 1 got 0
I20251026 02:56:27.770665 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 0cf76ef8f6af445eb0ce15c8d79136c5 (DEFAULT_TABLE table=default.Table_672f95f49551403a99355e82b17eaece [id=33334e177bb94b3e9de2a7e7ee59979b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.771078 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cf76ef8f6af445eb0ce15c8d79136c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.771773 2378 tablet_bootstrap.cc:492] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:27.772017 2378 ts_tablet_manager.cc:1403] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.161s user 0.004s sys 0.004s
I20251026 02:56:27.772689 1657 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 9d31527b9ceb4b17a55fd93aee79b9cd
I20251026 02:56:27.773168 2378 raft_consensus.cc:359] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.773450 2378 raft_consensus.cc:385] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.773572 2378 raft_consensus.cc:740] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.773888 2378 consensus_queue.cc:260] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.775336 2237 tablet_service.cc:1505] Processing CreateTablet for tablet f5228f482c0b4fcbbdf8ebc31bb62819 (DEFAULT_TABLE table=default.Table_0f53c2a8d63c47769dca3283cb871b9c [id=6300327f9d41469e9e70af59b7789ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.775799 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5228f482c0b4fcbbdf8ebc31bb62819. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.778585 2378 ts_tablet_manager.cc:1434] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251026 02:56:27.781517 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet cfed3523205548508d1101586ee3e2a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 44af74033ccf4d048475f12d164938f5) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.782228 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 7f22c6c163c34ed48a0b7eeb5a574e76 (DEFAULT_TABLE table=default.Table_0d7b9434a0494a618c25ceebd7f65de8 [id=bb7b13116ad94fa9bd15a1cdb15d207b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.782699 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f22c6c163c34ed48a0b7eeb5a574e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.785693 2889 tablet.cc:1722] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:27.789847 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet f437d8b5f5784e8fbec10d0700e90348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.791031 2890 tablet.cc:1722] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:27.791164 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 9a345fa5edbd48d2891b8b40b09cb3f8 (DEFAULT_TABLE table=default.Table_3d25c519f16647ca962f3862c6cd4f98 [id=eca037eee4b54221b7c9771c390055c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.791796 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a345fa5edbd48d2891b8b40b09cb3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.791877 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 4a3f3fc3c7a04d05bae28472251f3acc (DEFAULT_TABLE table=default.Table_840a6bd369ab416191c1f0ee15bd50ef [id=fae1b7f4879e4d338a4113383721fcc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.792397 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a3f3fc3c7a04d05bae28472251f3acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.800621 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 1fb2ff87cf0248a0a4a77f2733ef867b (DEFAULT_TABLE table=default.Table_26257eca63ce4baaa75beae2e9db1340 [id=fbc97817a9584474abc0b2ab2052abb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.801115 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fb2ff87cf0248a0a4a77f2733ef867b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.802268 1658 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]). Expected version 1 got 0
I20251026 02:56:27.803081 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e99391c4c0f74eb3ac3b62cedada8612" } modify_external_catalogs: true
I20251026 02:56:27.803272 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e99391c4c0f74eb3ac3b62cedada8612" } modify_external_catalogs: true
I20251026 02:56:27.804497 2376 tablet_replica.cc:333] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:27.804778 2376 raft_consensus.cc:2243] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:27.805006 2376 raft_consensus.cc:2272] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:27.806456 2377 ts_tablet_manager.cc:1929] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:27.806607 2377 log.cc:1199] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/461412f3ad1e4186b1302cdf5a624ed4
I20251026 02:56:27.806957 2377 ts_tablet_manager.cc:1950] T 461412f3ad1e4186b1302cdf5a624ed4 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:27.807267 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 288) took 1688 ms. Trace:
I20251026 02:56:27.807375 2377 rpcz_store.cc:276] 1026 02:56:26.119081 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.141705 (+ 22624us) service_pool.cc:225] Handling call
1026 02:56:27.529973 (+1388268us) ts_tablet_manager.cc:1151] Deleting tablet 461412f3ad1e4186b1302cdf5a624ed4
1026 02:56:27.529989 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:27.649550 (+119561us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:27.709464 (+ 59914us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:27.807238 (+ 97774us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1388033}
I20251026 02:56:27.807765 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 297) took 1533 ms. Trace:
I20251026 02:56:27.807862 2377 rpcz_store.cc:276] 1026 02:56:26.273841 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.273895 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:27.807637 (+1533742us) ts_tablet_manager.cc:1151] Deleting tablet 7801f073d5d94076af9299492f86fad2
1026 02:56:27.807645 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:27.807741 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1533411}
I20251026 02:56:27.808423 2376 ts_tablet_manager.cc:1916] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:27.809828 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 2524ee0c964345459c3e62f1e39b8f9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2524ee0c964345459c3e62f1e39b8f9e already in progress: creating tablet
I20251026 02:56:27.810329 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 7f22c6c163c34ed48a0b7eeb5a574e76 (DEFAULT_TABLE table=default.Table_0d7b9434a0494a618c25ceebd7f65de8 [id=bb7b13116ad94fa9bd15a1cdb15d207b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.810621 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 95c9fbae62db4850b0f02a2443a3432b (DEFAULT_TABLE table=default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.810840 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f22c6c163c34ed48a0b7eeb5a574e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.811087 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95c9fbae62db4850b0f02a2443a3432b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.812481 2378 tablet_bootstrap.cc:492] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:27.818259 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 1fb2ff87cf0248a0a4a77f2733ef867b (DEFAULT_TABLE table=default.Table_26257eca63ce4baaa75beae2e9db1340 [id=fbc97817a9584474abc0b2ab2052abb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.818655 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fb2ff87cf0248a0a4a77f2733ef867b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.819377 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 4a3f3fc3c7a04d05bae28472251f3acc (DEFAULT_TABLE table=default.Table_840a6bd369ab416191c1f0ee15bd50ef [id=fae1b7f4879e4d338a4113383721fcc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.819736 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a3f3fc3c7a04d05bae28472251f3acc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:27.823580 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2d6691bb4cf4a8292b0c082e54a91a5 consensus state has no leader
I20251026 02:56:27.836321 1661 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet a15339fe60924b76b8dc778d6c08a31f
I20251026 02:56:27.832679 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.850939 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9c7efb70c0804c149f36440a4a9e6ef8
I20251026 02:56:27.857843 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 5a00bc04b3654f90af342986751344e6 (DEFAULT_TABLE table=default.Table_af9006c41e484127a39c22ff0130d281 [id=91b4b1a8162441299c680e7c202a96d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.858382 1960 tablet_service.cc:1505] Processing CreateTablet for tablet d43b6f22dd4f4f9c9657f21514550312 (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.858834 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 41b7b0c36ba748f0a1565892dbe98a78 (DEFAULT_TABLE table=default.Table_292ad60362294da0976fd8fe6b180c0a [id=4651ecfc8aed4daf99de5a271e8afcda]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.859561 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c7efb70c0804c149f36440a4a9e6ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:27.858848 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a00bc04b3654f90af342986751344e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.859951 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41b7b0c36ba748f0a1565892dbe98a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.860484 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 4b5e61afe90e4bc4a2897de21e0b3919 (DEFAULT_TABLE table=default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.860706 1950 tablet_service.cc:1505] Processing CreateTablet for tablet c2d6691bb4cf4a8292b0c082e54a91a5 (DEFAULT_TABLE table=default.Table_8602d75889ea4576ae575b55fe95e940 [id=b0104213af32435589ae19109b7aee7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.860872 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b5e61afe90e4bc4a2897de21e0b3919. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.861094 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2d6691bb4cf4a8292b0c082e54a91a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.862529 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_15e626aa074c4736a61a03be25b64350" } new_table_name: "default.Table_cbef830fee8242fcad0278eb16b8e127" modify_external_catalogs: true
I20251026 02:56:27.863190 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 903e65fa8c214dc9a50cb97cab101d22 (table default.Table_862bd13f099645e8aafd987e4e850ae6 [id=65b5770c4b9243b892bfab963a439bb9]). Expected version 3 got 0
I20251026 02:56:27.863469 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 9a08e8b7eb1949408f2151f2d7257e9d (table default.Table_5f363b68bc2c4e7685f25e4eefa532a8 [id=43cb7f538f824ae38059645230cc7e94]). Expected version 3 got 0
I20251026 02:56:27.862450 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_065759ae67c2414ca2b73a3f68a4111c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.864333 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_065759ae67c2414ca2b73a3f68a4111c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.865873 2631 consensus_queue.cc:1048] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:27.866392 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9f48bbcff2cc4a47b64cb3862caccc33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.866477 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0ea323ca655f45aa94b47eab713ebbd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.866822 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a7b1603151c54757bd2505c7958e3bae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:27.867204 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet a378e913112c443db869016e3b98fd0e from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:27.867291 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7b1603151c54757bd2505c7958e3bae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.868736 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 461412f3ad1e4186b1302cdf5a624ed4 (table default.Table_095bb281b8804e38b5bd56a1dbabacfa [id=6cab490620ef48f18ab808cd5d6c6af8]) successfully deleted
W20251026 02:56:27.869356 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ea323ca655f45aa94b47eab713ebbd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.870699 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a37eaaa2d8244f6f95a762d240e1dffb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:27.871245 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a37eaaa2d8244f6f95a762d240e1dffb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:27.871520 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f48bbcff2cc4a47b64cb3862caccc33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:27.874221 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 41b7b0c36ba748f0a1565892dbe98a78 (DEFAULT_TABLE table=default.Table_292ad60362294da0976fd8fe6b180c0a [id=4651ecfc8aed4daf99de5a271e8afcda]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.874441 2241 tablet_service.cc:1505] Processing CreateTablet for tablet d43b6f22dd4f4f9c9657f21514550312 (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.874696 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41b7b0c36ba748f0a1565892dbe98a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.874784 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c7efb70c0804c149f36440a4a9e6ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:27.875070 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43b6f22dd4f4f9c9657f21514550312. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.880204 2758 raft_consensus.cc:493] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:27.882102 2891 raft_consensus.cc:493] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:27.882333 2891 raft_consensus.cc:515] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.884296 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c7efb70c0804c149f36440a4a9e6ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:27.884317 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 299) took 1524 ms. Trace:
W20251026 02:56:27.884446 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d212170f056a44dfb15cfcb7f9fbae2f consensus state has no leader
I20251026 02:56:27.884487 2089 tablet_service.cc:1505] Processing CreateTablet for tablet c2d6691bb4cf4a8292b0c082e54a91a5 (DEFAULT_TABLE table=default.Table_8602d75889ea4576ae575b55fe95e940 [id=b0104213af32435589ae19109b7aee7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.884450 2377 rpcz_store.cc:276] 1026 02:56:26.283441 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.283496 (+ 55us) service_pool.cc:225] Handling call
1026 02:56:27.808077 (+1524581us) ts_tablet_manager.cc:1151] Deleting tablet a70d70abc3a34dac8cf9a747903fa78c
1026 02:56:27.808085 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:27.808181 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1524294}
I20251026 02:56:27.884622 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 95c9fbae62db4850b0f02a2443a3432b (DEFAULT_TABLE table=default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.884832 2377 tablet_replica.cc:333] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:27.885174 2377 raft_consensus.cc:2243] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:27.885568 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2d6691bb4cf4a8292b0c082e54a91a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.885586 2377 raft_consensus.cc:2272] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:27.885828 2891 leader_election.cc:290] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:27.886117 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e5bb0a6a9f947cf9ce119c1fc91fbf5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:27.886539 2377 ts_tablet_manager.cc:1916] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:27.886705 2194 leader_election.cc:343] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.886761 2372 ts_tablet_manager.cc:1929] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:27.887161 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95c9fbae62db4850b0f02a2443a3432b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.887650 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 95c9fbae62db4850b0f02a2443a3432b (DEFAULT_TABLE table=default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.888443 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 7f22c6c163c34ed48a0b7eeb5a574e76 (DEFAULT_TABLE table=default.Table_0d7b9434a0494a618c25ceebd7f65de8 [id=bb7b13116ad94fa9bd15a1cdb15d207b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:27.888499 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 58413.1us
I20251026 02:56:27.888697 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 4b5e61afe90e4bc4a2897de21e0b3919 (DEFAULT_TABLE table=default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.888967 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f22c6c163c34ed48a0b7eeb5a574e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.888890 2758 raft_consensus.cc:515] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.889740 2165 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.889801 2093 tablet_service.cc:1505] Processing CreateTablet for tablet d43b6f22dd4f4f9c9657f21514550312 (DEFAULT_TABLE table=default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.890089 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 41b7b0c36ba748f0a1565892dbe98a78 (DEFAULT_TABLE table=default.Table_292ad60362294da0976fd8fe6b180c0a [id=4651ecfc8aed4daf99de5a271e8afcda]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.890654 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43b6f22dd4f4f9c9657f21514550312. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.892489 2758 leader_election.cc:290] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:27.892303 2098 tablet_service.cc:1505] Processing CreateTablet for tablet c0bacd15b41149b5856d20068711b432 (DEFAULT_TABLE table=default.Table_a1e745ce0f454f4fa2732de1282afae1 [id=188c051bbce94f7489008daaa95529b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.893194 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 5a00bc04b3654f90af342986751344e6 (DEFAULT_TABLE table=default.Table_af9006c41e484127a39c22ff0130d281 [id=91b4b1a8162441299c680e7c202a96d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.894096 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 1fb2ff87cf0248a0a4a77f2733ef867b (DEFAULT_TABLE table=default.Table_26257eca63ce4baaa75beae2e9db1340 [id=fbc97817a9584474abc0b2ab2052abb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.894639 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22e4a433801a4cb58062d3e76c760b69" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:27.895354 1654 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]). Expected version 1 got 0
I20251026 02:56:27.895779 2894 raft_consensus.cc:493] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:27.895925 2894 raft_consensus.cc:515] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.897195 2894 leader_election.cc:290] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:27.897682 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9156314f98ac4ed9981f6f13fc585043" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:27.898243 2194 leader_election.cc:343] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.900534 2381 tablet_bootstrap.cc:654] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:27.901623 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7801f073d5d94076af9299492f86fad2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7801f073d5d94076af9299492f86fad2 already in progress: creating tablet
I20251026 02:56:27.901826 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet 8c1e63ac8c5645f3ba1578792c0a0dae from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:27.903120 2378 tablet_bootstrap.cc:654] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:27.903136 2194 consensus_peers.cc:597] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:27.904600 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet a70d70abc3a34dac8cf9a747903fa78c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a70d70abc3a34dac8cf9a747903fa78c already in progress: creating tablet
W20251026 02:56:27.907420 2194 consensus_peers.cc:597] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:27.908087 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22e4a433801a4cb58062d3e76c760b69" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:27.909039 1954 tablet_service.cc:1505] Processing CreateTablet for tablet c0bacd15b41149b5856d20068711b432 (DEFAULT_TABLE table=default.Table_a1e745ce0f454f4fa2732de1282afae1 [id=188c051bbce94f7489008daaa95529b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.909863 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0bacd15b41149b5856d20068711b432. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.913710 2236 tablet_service.cc:1505] Processing CreateTablet for tablet c2d6691bb4cf4a8292b0c082e54a91a5 (DEFAULT_TABLE table=default.Table_8602d75889ea4576ae575b55fe95e940 [id=b0104213af32435589ae19109b7aee7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.914189 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2d6691bb4cf4a8292b0c082e54a91a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.914361 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95c9fbae62db4850b0f02a2443a3432b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.890321 2372 log.cc:1199] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/4a5343ba771a4e8ea2f467c4c619c1f9
I20251026 02:56:27.921514 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.922132 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 5a00bc04b3654f90af342986751344e6 (DEFAULT_TABLE table=default.Table_af9006c41e484127a39c22ff0130d281 [id=91b4b1a8162441299c680e7c202a96d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.924108 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 4b5e61afe90e4bc4a2897de21e0b3919 (DEFAULT_TABLE table=default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.924707 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41b7b0c36ba748f0a1565892dbe98a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.924983 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b5e61afe90e4bc4a2897de21e0b3919. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:27.925726 2195 consensus_peers.cc:597] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:27.926863 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fb2ff87cf0248a0a4a77f2733ef867b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.928354 2381 tablet_bootstrap.cc:492] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:27.928567 2381 ts_tablet_manager.cc:1403] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.170s user 0.004s sys 0.000s
W20251026 02:56:27.928637 2053 leader_election.cc:343] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.929517 2375 tablet_bootstrap.cc:492] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:27.931365 2115 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9156314f98ac4ed9981f6f13fc585043" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:27.931576 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0bacd15b41149b5856d20068711b432. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.933127 2381 raft_consensus.cc:359] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.933332 2381 raft_consensus.cc:385] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:27.933391 2381 raft_consensus.cc:740] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:27.933604 2381 consensus_queue.cc:260] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.931035 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b5e61afe90e4bc4a2897de21e0b3919. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.935998 2372 ts_tablet_manager.cc:1950] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:27.936304 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a00bc04b3654f90af342986751344e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.935007 2902 tablet.cc:1722] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:27.936661 2115 raft_consensus.cc:2468] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:27.936880 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43b6f22dd4f4f9c9657f21514550312. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.939723 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 4a5343ba771a4e8ea2f467c4c619c1f9 (table default.Table_7296727c5b2f4c95ac3ff62818bd84f2 [id=af7a0605122745588d028814d3583cc6]) successfully deleted
W20251026 02:56:27.940063 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 155a23a93358487a9564bb5ac175b4eb because the tablet was not found. No further retry: Not found: Tablet not found: 155a23a93358487a9564bb5ac175b4eb
I20251026 02:56:27.941824 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a00bc04b3654f90af342986751344e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.942265 2372 tablet_replica.cc:333] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:27.943306 2372 raft_consensus.cc:2243] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:27.932108 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 4a3f3fc3c7a04d05bae28472251f3acc (DEFAULT_TABLE table=default.Table_840a6bd369ab416191c1f0ee15bd50ef [id=fae1b7f4879e4d338a4113383721fcc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:27.945906 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a3f3fc3c7a04d05bae28472251f3acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:27.946496 1655 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_424693529d0d4c829755adc2c9835c0e [id=690ec15818d34cdfa990c8fcd36a19cd] alter complete (version 1)
I20251026 02:56:27.948719 2372 raft_consensus.cc:2272] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:27.956359 2372 ts_tablet_manager.cc:1916] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:27.961771 2631 consensus_queue.cc:1048] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251026 02:56:27.962270 2376 ts_tablet_manager.cc:1929] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:27.963536 2894 raft_consensus.cc:493] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:27.965080 2894 raft_consensus.cc:515] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.965945 2894 leader_election.cc:290] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:27.969097 2381 ts_tablet_manager.cc:1434] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20251026 02:56:27.976444 2195 leader_election.cc:304] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:27.978433 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 22e4a433801a4cb58062d3e76c760b69 (table default.Table_f7c216f3ebd34c718ceef042a435b4cd [id=b4c0611288a949378eed16d32b5170b9]). Expected version 2 got 0
I20251026 02:56:27.978717 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]). Expected version 1 got 0
I20251026 02:56:27.988317 2894 raft_consensus.cc:2804] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:27.988441 2894 raft_consensus.cc:493] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:27.988559 2894 raft_consensus.cc:3060] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:27.987890 2308 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:27.993654 2894 raft_consensus.cc:515] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:27.995512 2894 leader_election.cc:290] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:27.995602 2053 leader_election.cc:343] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:27.995772 2053 leader_election.cc:304] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:27.996142 2824 raft_consensus.cc:2749] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:27.996203 2375 ts_tablet_manager.cc:1403] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.280s user 0.003s sys 0.005s
I20251026 02:56:27.997655 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8533f1c191042ee9f2c0b99972fe931" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:27.997969 2117 raft_consensus.cc:2468] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:27.996140 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 66671us
I20251026 02:56:27.966804 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8533f1c191042ee9f2c0b99972fe931" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:27.995227 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9156314f98ac4ed9981f6f13fc585043" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:28.009758 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 122de8616b8f4fcf88971d5738c47e82 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:28.009814 2194 leader_election.cc:343] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:28.010064 2194 leader_election.cc:343] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:28.012064 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8d1fb0a5abc3483e85ee11cfc62a0b13" } new_table_name: "default.Table_df797fe86da84688840a0e8a2d2f18ed" modify_external_catalogs: true
I20251026 02:56:28.012450 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_133a36cbb31242b680078785eb21866e" } new_table_name: "default.Table_c2ef795e86444f9f85bb7d0bcca51240" modify_external_catalogs: true
I20251026 02:56:28.013748 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f4fc99282b5643e89641bb7082d165a5" } modify_external_catalogs: true
I20251026 02:56:28.013892 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f4fc99282b5643e89641bb7082d165a5" } modify_external_catalogs: true
I20251026 02:56:28.014549 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9f3815b1bf464a83a574e3075d065956" } modify_external_catalogs: true
I20251026 02:56:28.014715 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9f3815b1bf464a83a574e3075d065956" } modify_external_catalogs: true
I20251026 02:56:28.015213 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8611fe991ff441caab2cd1faa546ddea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.015758 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8611fe991ff441caab2cd1faa546ddea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.017364 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6d91747dd6164f0d84509497a93aeeea" } modify_external_catalogs: true
I20251026 02:56:28.017562 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6d91747dd6164f0d84509497a93aeeea" } modify_external_catalogs: true
I20251026 02:56:28.019167 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ecfe7bd401bb42f2bcdcbb956e55779d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.019797 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecfe7bd401bb42f2bcdcbb956e55779d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.021147 2381 tablet_bootstrap.cc:492] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:28.021883 2243 tablet_service.cc:1505] Processing CreateTablet for tablet c0bacd15b41149b5856d20068711b432 (DEFAULT_TABLE table=default.Table_a1e745ce0f454f4fa2732de1282afae1 [id=188c051bbce94f7489008daaa95529b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.022325 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0bacd15b41149b5856d20068711b432. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.022652 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_73e25969b3304d3a8d466a21f9445fcb" } new_table_name: "default.Table_1818605e0bc548938337de5d5fc7a781" modify_external_catalogs: true
I20251026 02:56:27.962404 2376 log.cc:1199] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/0020cb1f190c49978c5322d387a47f71
I20251026 02:56:28.024294 2376 ts_tablet_manager.cc:1950] T 0020cb1f190c49978c5322d387a47f71 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:28.028786 2195 leader_election.cc:304] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:28.029397 2195 leader_election.cc:343] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9e5bb0a6a9f947cf9ce119c1fc91fbf5
I20251026 02:56:28.029626 2195 leader_election.cc:304] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:28.029767 2891 raft_consensus.cc:2804] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:28.029903 2891 raft_consensus.cc:493] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:28.029935 2440 raft_consensus.cc:2749] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:28.030002 2891 raft_consensus.cc:3060] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:28.030678 2194 consensus_peers.cc:597] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:28.033211 2891 raft_consensus.cc:515] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.033869 2891 leader_election.cc:290] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:28.034319 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8533f1c191042ee9f2c0b99972fe931" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:28.035063 2194 leader_election.cc:343] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:28.035392 2897 raft_consensus.cc:493] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:28.035598 2897 raft_consensus.cc:515] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.037292 2375 raft_consensus.cc:359] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.038399 2117 raft_consensus.cc:1275] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:28.039592 2891 consensus_queue.cc:1048] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:28.041550 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9156314f98ac4ed9981f6f13fc585043" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
W20251026 02:56:28.045434 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efd751ee4b1d433198dcd00c081f648d consensus state has no leader
I20251026 02:56:28.046859 2381 tablet_bootstrap.cc:654] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.047294 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ff6f3f84ac2348b79403d10fc9e3ecb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.048020 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff6f3f84ac2348b79403d10fc9e3ecb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.050093 2377 ts_tablet_manager.cc:1929] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:28.050372 2377 log.cc:1199] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/8667ffa25f2b478a987826d486400bbe
I20251026 02:56:28.050846 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa399e6407fd4c4d9871d3d5ccf894ec" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:28.052048 2377 ts_tablet_manager.cc:1950] T 8667ffa25f2b478a987826d486400bbe P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:28.051985 2902 tablet.cc:1722] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:28.052598 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 300) took 1766 ms. Trace:
I20251026 02:56:28.052798 2377 rpcz_store.cc:276] 1026 02:56:26.285878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.285938 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:27.884749 (+1598811us) ts_tablet_manager.cc:1151] Deleting tablet 8667ffa25f2b478a987826d486400bbe
1026 02:56:27.884758 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:27.961509 (+ 76751us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.049948 (+ 88439us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.052551 (+ 2603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598598}
I20251026 02:56:28.053161 2376 tablet_replica.cc:333] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:28.053194 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8533f1c191042ee9f2c0b99972fe931" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:28.054157 2376 raft_consensus.cc:2243] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:28.040364 2899 tablet.cc:1722] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:28.053875 2378 tablet_bootstrap.cc:492] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:28.065625 2375 raft_consensus.cc:385] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.065997 2378 ts_tablet_manager.cc:1403] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.254s user 0.003s sys 0.003s
I20251026 02:56:28.066313 2117 raft_consensus.cc:3060] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:28.067165 2053 leader_election.cc:343] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:28.067673 2118 raft_consensus.cc:3060] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:28.068776 2378 raft_consensus.cc:359] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.069093 2378 raft_consensus.cc:385] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.069243 2378 raft_consensus.cc:740] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.069597 2378 consensus_queue.cc:260] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.070462 2378 ts_tablet_manager.cc:1434] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:28.071790 2117 raft_consensus.cc:2468] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:28.073530 2378 tablet_bootstrap.cc:492] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:28.074324 2376 raft_consensus.cc:2272] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:28.076530 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 0020cb1f190c49978c5322d387a47f71 (table default.Table_689f802c9e4b457eb71fb78898fe14b0 [id=05242557de9a4a19948b8591ce73ab28]) successfully deleted
I20251026 02:56:28.079236 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a2e4dfe2f01f4d8fac1b89211f87d9a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.079811 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2e4dfe2f01f4d8fac1b89211f87d9a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.082417 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 86e483c52e514f1da54335479f59d68e
I20251026 02:56:28.082983 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet 8dbc12d49b094dbeadad2e76916ba9af from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:28.070192 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa399e6407fd4c4d9871d3d5ccf894ec" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:28.070502 2889 tablet.cc:1722] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:28.089570 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 461412f3ad1e4186b1302cdf5a624ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:28.089936 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 86e483c52e514f1da54335479f59d68e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:28.093945 2375 raft_consensus.cc:740] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.093715 2897 leader_election.cc:290] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:28.096568 2378 tablet_bootstrap.cc:654] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.072350 2195 leader_election.cc:304] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:28.097707 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_503013f4998040b8a8d436174cfef014"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.098387 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_503013f4998040b8a8d436174cfef014 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.098724 2894 raft_consensus.cc:2804] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:28.098927 2894 raft_consensus.cc:697] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:28.099283 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 86e483c52e514f1da54335479f59d68e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:28.099249 2894 consensus_queue.cc:237] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.101311 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fa634331e4bd4c14a102908bc41edba1" } new_table_name: "default.Table_c9c7b9787c3a458ea41be132b0841599" modify_external_catalogs: true
I20251026 02:56:28.072759 2911 tablet.cc:1722] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:28.102468 2375 consensus_queue.cc:260] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.103080 2375 ts_tablet_manager.cc:1434] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.107s user 0.002s sys 0.000s
W20251026 02:56:28.105579 2194 consensus_peers.cc:597] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:28.109941 2377 tablet_replica.cc:333] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539: stopping tablet replica
W20251026 02:56:28.110651 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b9a5daa36b547a7888d38dc3329eaf2 has no consensus state
I20251026 02:56:28.110872 2377 raft_consensus.cc:2243] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:28.110282 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 8667ffa25f2b478a987826d486400bbe (table default.Table_4fd5f39751624015ad57dd7fc77ce18a [id=f7f1e852a46f4129b2709fd58a6be034]) successfully deleted
I20251026 02:56:28.111248 2377 raft_consensus.cc:2272] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:28.112489 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:28.114763 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_feb822201306444ebe478734d286a667" } modify_external_catalogs: true
I20251026 02:56:28.114992 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_feb822201306444ebe478734d286a667" } modify_external_catalogs: true
I20251026 02:56:28.118602 2377 ts_tablet_manager.cc:1916] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:28.123332 2053 leader_election.cc:343] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:28.125123 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 86e483c52e514f1da54335479f59d68e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:28.128310 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 049a915b35224cdaa38df77c734a0f01 consensus state has no leader
I20251026 02:56:28.132706 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_42c4c5e055054536b2b541711288a481" } modify_external_catalogs: true
I20251026 02:56:28.133011 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_42c4c5e055054536b2b541711288a481" } modify_external_catalogs: true
I20251026 02:56:28.135730 2308 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:28.135875 2375 tablet_bootstrap.cc:492] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:28.125370 2053 leader_election.cc:304] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:28.143038 2897 raft_consensus.cc:2749] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:28.147677 2381 tablet_bootstrap.cc:492] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:28.147867 2381 ts_tablet_manager.cc:1403] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.127s user 0.004s sys 0.000s
I20251026 02:56:28.149549 2381 raft_consensus.cc:359] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.149772 2381 raft_consensus.cc:385] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.149873 2381 raft_consensus.cc:740] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.155055 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 340bf8c78a88413299b61a52f0a578f3 (table default.Table_1361fbf89aa04321a7876c1e43e27089 [id=c162ef89d2f14e91a24a6d8c2c37c261]). Expected version 4 got 0
I20251026 02:56:28.156687 2377 ts_tablet_manager.cc:1929] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:28.161000 2115 raft_consensus.cc:1275] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:28.164178 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 115ccccf16b74efc9f24b607cf0c74d1 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:28.161618 2759 consensus_queue.cc:1048] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:28.165251 2377 log.cc:1199] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/a2bfbe624075463f8b90ed4ec66b784c
I20251026 02:56:28.165568 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3c56d6d5af8c44139ac28b8cc9945a9f" } new_table_name: "default.Table_a59579db048a4eec8a0a47544626d25f" modify_external_catalogs: true
I20251026 02:56:28.166030 2377 ts_tablet_manager.cc:1950] T a2bfbe624075463f8b90ed4ec66b784c P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:28.166419 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 301) took 1674 ms. Trace:
I20251026 02:56:28.166538 2377 rpcz_store.cc:276] 1026 02:56:26.492274 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.492333 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:28.109852 (+1617519us) ts_tablet_manager.cc:1151] Deleting tablet a2bfbe624075463f8b90ed4ec66b784c
1026 02:56:28.109864 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:28.141340 (+ 31476us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.156544 (+ 15204us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.166381 (+ 9837us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1617275}
I20251026 02:56:28.167054 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet a2bfbe624075463f8b90ed4ec66b784c (table default.Table_3792bcbbc16b4be0b113cb5f07b9261b [id=8a3d1c20763445c499bd655eb99deb56]) successfully deleted
I20251026 02:56:28.169241 2377 tablet_replica.cc:333] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:28.169448 2377 raft_consensus.cc:2243] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:28.169658 2377 raft_consensus.cc:2272] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:28.172547 2911 tablet.cc:1722] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:28.174062 2378 tablet_bootstrap.cc:492] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:28.174250 2378 ts_tablet_manager.cc:1403] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.101s user 0.013s sys 0.022s
I20251026 02:56:28.175599 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4293bc067ecf4838a1557178f2c53ec1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.176258 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4293bc067ecf4838a1557178f2c53ec1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.176508 2372 ts_tablet_manager.cc:1929] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:28.177623 2372 log.cc:1199] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/81530bbf4a2d4e908add683475e594fb
I20251026 02:56:28.177760 2377 ts_tablet_manager.cc:1916] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:28.177722 2378 raft_consensus.cc:359] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.177983 2378 raft_consensus.cc:385] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.178063 2378 raft_consensus.cc:740] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.178356 2378 consensus_queue.cc:260] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.178858 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:28.180014 2372 ts_tablet_manager.cc:1950] T 81530bbf4a2d4e908add683475e594fb P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:28.184598 2899 tablet.cc:1722] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:28.188112 2165 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:28.188570 2375 tablet_bootstrap.cc:654] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.188660 1663 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (table default.Table_c9c7b9787c3a458ea41be132b0841599 [id=1d09f5b9fa164ac783f0be9332eac4c3]). Expected version 5 got 0
I20251026 02:56:28.189397 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 03da415f3a944a9a9679b3a45295ce35 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:28.191640 2378 ts_tablet_manager.cc:1434] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251026 02:56:28.192041 2378 tablet_bootstrap.cc:492] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:28.193409 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 1eabfc95caa84afc9a8956a7ac13fea6 (DEFAULT_TABLE table=default.Table_0ea323ca655f45aa94b47eab713ebbd0 [id=1d82a5d03031489fa756df9cfb380b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.193876 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eabfc95caa84afc9a8956a7ac13fea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.193933 2378 tablet_bootstrap.cc:654] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.195159 2118 raft_consensus.cc:2468] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:28.195333 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 22e4a433801a4cb58062d3e76c760b69 (table default.Table_f7c216f3ebd34c718ceef042a435b4cd [id=b4c0611288a949378eed16d32b5170b9]). Expected version 2 got 0
I20251026 02:56:28.195487 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:28.197161 2195 leader_election.cc:304] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:28.197923 2440 raft_consensus.cc:2804] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:28.198045 2440 raft_consensus.cc:697] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:28.198289 2440 consensus_queue.cc:237] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.200491 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 81530bbf4a2d4e908add683475e594fb (table default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]) successfully deleted
W20251026 02:56:28.200788 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ced7ad6a601d432cbc13b3199a0506e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ced7ad6a601d432cbc13b3199a0506e4 already in progress: creating tablet
I20251026 02:56:28.200937 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 1eabfc95caa84afc9a8956a7ac13fea6 (DEFAULT_TABLE table=default.Table_0ea323ca655f45aa94b47eab713ebbd0 [id=1d82a5d03031489fa756df9cfb380b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.201398 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eabfc95caa84afc9a8956a7ac13fea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.201331 1960 tablet_service.cc:1505] Processing CreateTablet for tablet f2d97084b5914350afbfda5c2a34bf9a (DEFAULT_TABLE table=default.Table_a7b1603151c54757bd2505c7958e3bae [id=cd7cfcd0eac044609fe5b2751ac6a98a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.201730 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d97084b5914350afbfda5c2a34bf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.205071 2376 ts_tablet_manager.cc:1916] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:28.207649 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 1eabfc95caa84afc9a8956a7ac13fea6 (DEFAULT_TABLE table=default.Table_0ea323ca655f45aa94b47eab713ebbd0 [id=1d82a5d03031489fa756df9cfb380b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.210286 2372 tablet_replica.cc:333] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:28.210603 2372 raft_consensus.cc:2243] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:28.212481 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 47037db3602246a2ab6224557286162f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47037db3602246a2ab6224557286162f already in progress: creating tablet
I20251026 02:56:28.212754 2372 raft_consensus.cc:2272] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:28.213774 1655 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_862bd13f099645e8aafd987e4e850ae6 [id=65b5770c4b9243b892bfab963a439bb9] alter complete (version 3)
I20251026 02:56:28.214692 2372 ts_tablet_manager.cc:1916] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:28.208189 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eabfc95caa84afc9a8956a7ac13fea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.218788 2093 tablet_service.cc:1505] Processing CreateTablet for tablet f2d97084b5914350afbfda5c2a34bf9a (DEFAULT_TABLE table=default.Table_a7b1603151c54757bd2505c7958e3bae [id=cd7cfcd0eac044609fe5b2751ac6a98a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.219267 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d97084b5914350afbfda5c2a34bf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.221568 2232 tablet_service.cc:1505] Processing CreateTablet for tablet f2d97084b5914350afbfda5c2a34bf9a (DEFAULT_TABLE table=default.Table_a7b1603151c54757bd2505c7958e3bae [id=cd7cfcd0eac044609fe5b2751ac6a98a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.222002 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d97084b5914350afbfda5c2a34bf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.221943 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (table default.Table_c9c7b9787c3a458ea41be132b0841599 [id=1d09f5b9fa164ac783f0be9332eac4c3]). Expected version 5 got 0
I20251026 02:56:28.223086 2661 raft_consensus.cc:493] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:28.223294 2661 raft_consensus.cc:515] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.224364 2661 leader_election.cc:290] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:28.225150 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 1dbbfb77851d443f8cf4d798f3e07e59 (DEFAULT_TABLE table=default.Table_065759ae67c2414ca2b73a3f68a4111c [id=b7f8d9dfa47b4c499af7d0e09d417c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.225965 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_de67223546af408ead7b2d0d3979d78d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.226559 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de67223546af408ead7b2d0d3979d78d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.228286 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78a624d839934d3089b64e397d4360af" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:28.228415 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_60d87251cafc4e6ca68c5e0affce8260"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.229022 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60d87251cafc4e6ca68c5e0affce8260 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.232985 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dbbfb77851d443f8cf4d798f3e07e59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.240454 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 1dbbfb77851d443f8cf4d798f3e07e59 (DEFAULT_TABLE table=default.Table_065759ae67c2414ca2b73a3f68a4111c [id=b7f8d9dfa47b4c499af7d0e09d417c9b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:28.241601 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4b0933f71cb4ac7a58986d16b613e64 consensus state has no leader
I20251026 02:56:28.241901 2091 tablet_service.cc:1505] Processing CreateTablet for tablet cfac07917ee34e6dba3fac5e081f6b43 (DEFAULT_TABLE table=default.Table_5e91141bf6d441ab9d1842e8a859869a [id=a8d490f3dbc948d38c54d14e178b736e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.242323 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfac07917ee34e6dba3fac5e081f6b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.243124 2232 tablet_service.cc:1505] Processing CreateTablet for tablet cfac07917ee34e6dba3fac5e081f6b43 (DEFAULT_TABLE table=default.Table_5e91141bf6d441ab9d1842e8a859869a [id=a8d490f3dbc948d38c54d14e178b736e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.243275 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 41a1442fffe54add93666e3c78a0cc58 (DEFAULT_TABLE table=default.Table_0dfae834c6294764ae2c58c6a11c5280 [id=a106f9ece0b9467a8bc1d5d4b3524c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.242794 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8c3d1109382545ffa157d6a3cb39f322"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:28.243710 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41a1442fffe54add93666e3c78a0cc58. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:28.243827 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c3d1109382545ffa157d6a3cb39f322 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.244290 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dbbfb77851d443f8cf4d798f3e07e59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.245067 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfac07917ee34e6dba3fac5e081f6b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.246125 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 1dbbfb77851d443f8cf4d798f3e07e59 (DEFAULT_TABLE table=default.Table_065759ae67c2414ca2b73a3f68a4111c [id=b7f8d9dfa47b4c499af7d0e09d417c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.246505 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dbbfb77851d443f8cf4d798f3e07e59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.252471 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 41a1442fffe54add93666e3c78a0cc58 (DEFAULT_TABLE table=default.Table_0dfae834c6294764ae2c58c6a11c5280 [id=a106f9ece0b9467a8bc1d5d4b3524c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.252898 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41a1442fffe54add93666e3c78a0cc58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.255481 2086 tablet_service.cc:1505] Processing CreateTablet for tablet e89a98fcfdbb4011876b0ebc2af91f67 (DEFAULT_TABLE table=default.Table_5a3412a706124204a0379462d7628a17 [id=881224ae789e4daab936faafe4765729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.255909 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e89a98fcfdbb4011876b0ebc2af91f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.258953 2087 tablet_service.cc:1505] Processing CreateTablet for tablet dfd3e23d83a84648bd11cd258377aecb (DEFAULT_TABLE table=default.Table_f4bddfca37d64369b99a1fdc0102ca35 [id=434c941e53194bcf9a53a590775d6a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.259379 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfd3e23d83a84648bd11cd258377aecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.260111 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 398c1f420974453b9b6b0fe17795378b (DEFAULT_TABLE table=default.Table_461dd114dc774ff7bb820f85a2acc574 [id=7f64fbec3582415a8924d01e14856a56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.260571 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 398c1f420974453b9b6b0fe17795378b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.261173 2243 tablet_service.cc:1505] Processing CreateTablet for tablet dfd3e23d83a84648bd11cd258377aecb (DEFAULT_TABLE table=default.Table_f4bddfca37d64369b99a1fdc0102ca35 [id=434c941e53194bcf9a53a590775d6a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.261550 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfd3e23d83a84648bd11cd258377aecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.263619 2372 ts_tablet_manager.cc:1929] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:28.263743 2372 log.cc:1199] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/c570e5f9a93c45bda32ec92c68794714
I20251026 02:56:28.264761 2372 ts_tablet_manager.cc:1950] T c570e5f9a93c45bda32ec92c68794714 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:28.265451 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet c570e5f9a93c45bda32ec92c68794714 (table default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]) successfully deleted
W20251026 02:56:28.265894 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 2524ee0c964345459c3e62f1e39b8f9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2524ee0c964345459c3e62f1e39b8f9e already in progress: creating tablet
W20251026 02:56:28.266199 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ec13c5386abe414ba25589e03d5b850a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec13c5386abe414ba25589e03d5b850a already in progress: creating tablet
I20251026 02:56:28.267283 2092 tablet_service.cc:1505] Processing CreateTablet for tablet e203fe0403594d30b44d531153d42a10 (DEFAULT_TABLE table=default.Table_874fdf4c2b4e4ace9b87c72de1e573a7 [id=4226c45f4f204b3ca19dd2e0c7a15fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.267715 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e203fe0403594d30b44d531153d42a10. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:28.268039 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 741792cf96164695b03de3fb86b12602 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 741792cf96164695b03de3fb86b12602 already in progress: creating tablet
I20251026 02:56:28.269634 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 398c1f420974453b9b6b0fe17795378b (DEFAULT_TABLE table=default.Table_461dd114dc774ff7bb820f85a2acc574 [id=7f64fbec3582415a8924d01e14856a56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.270172 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 398c1f420974453b9b6b0fe17795378b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.270689 2911 tablet.cc:1722] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:28.272544 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 9b9a5daa36b547a7888d38dc3329eaf2 (DEFAULT_TABLE table=default.Table_df797fe86da84688840a0e8a2d2f18ed [id=26a5eae9586f4d00b5ebaf9eb3462272]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.272989 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9a5daa36b547a7888d38dc3329eaf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.273295 2089 tablet_service.cc:1505] Processing CreateTablet for tablet 9b9a5daa36b547a7888d38dc3329eaf2 (DEFAULT_TABLE table=default.Table_df797fe86da84688840a0e8a2d2f18ed [id=26a5eae9586f4d00b5ebaf9eb3462272]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.273676 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9a5daa36b547a7888d38dc3329eaf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.269634 2231 tablet_service.cc:1505] Processing CreateTablet for tablet e203fe0403594d30b44d531153d42a10 (DEFAULT_TABLE table=default.Table_874fdf4c2b4e4ace9b87c72de1e573a7 [id=4226c45f4f204b3ca19dd2e0c7a15fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.278461 2236 tablet_service.cc:1505] Processing CreateTablet for tablet e89a98fcfdbb4011876b0ebc2af91f67 (DEFAULT_TABLE table=default.Table_5a3412a706124204a0379462d7628a17 [id=881224ae789e4daab936faafe4765729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.278992 2902 tablet.cc:1722] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:28.280898 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 9fb42b54f558499090a1d7cf257dd84c (DEFAULT_TABLE table=default.Table_8fbaf8dac35d47848025cbb47c261c08 [id=9ac57a0fe65c4e07b113f83dacca8043]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.281333 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fb42b54f558499090a1d7cf257dd84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.281539 2098 tablet_service.cc:1505] Processing CreateTablet for tablet a53f25688e3b45dca68984e2c543d094 (DEFAULT_TABLE table=default.Table_feb822201306444ebe478734d286a667 [id=535d3143b66249a083853a11934d994e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.282222 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53f25688e3b45dca68984e2c543d094. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.284052 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e203fe0403594d30b44d531153d42a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.284272 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c1abfb93d24a4f2a97d8d87a06c34ea5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.284813 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1abfb93d24a4f2a97d8d87a06c34ea5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.288630 2242 tablet_service.cc:1505] Processing CreateTablet for tablet a53f25688e3b45dca68984e2c543d094 (DEFAULT_TABLE table=default.Table_feb822201306444ebe478734d286a667 [id=535d3143b66249a083853a11934d994e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.290114 1955 tablet_service.cc:1505] Processing CreateTablet for tablet cfac07917ee34e6dba3fac5e081f6b43 (DEFAULT_TABLE table=default.Table_5e91141bf6d441ab9d1842e8a859869a [id=a8d490f3dbc948d38c54d14e178b736e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.291517 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 9fb42b54f558499090a1d7cf257dd84c (DEFAULT_TABLE table=default.Table_8fbaf8dac35d47848025cbb47c261c08 [id=9ac57a0fe65c4e07b113f83dacca8043]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.293663 1950 tablet_service.cc:1505] Processing CreateTablet for tablet e89a98fcfdbb4011876b0ebc2af91f67 (DEFAULT_TABLE table=default.Table_5a3412a706124204a0379462d7628a17 [id=881224ae789e4daab936faafe4765729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.294169 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 398c1f420974453b9b6b0fe17795378b (DEFAULT_TABLE table=default.Table_461dd114dc774ff7bb820f85a2acc574 [id=7f64fbec3582415a8924d01e14856a56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.294482 1958 tablet_service.cc:1505] Processing CreateTablet for tablet dfd3e23d83a84648bd11cd258377aecb (DEFAULT_TABLE table=default.Table_f4bddfca37d64369b99a1fdc0102ca35 [id=434c941e53194bcf9a53a590775d6a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.294726 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 41a1442fffe54add93666e3c78a0cc58 (DEFAULT_TABLE table=default.Table_0dfae834c6294764ae2c58c6a11c5280 [id=a106f9ece0b9467a8bc1d5d4b3524c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.295086 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfac07917ee34e6dba3fac5e081f6b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.296801 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 398c1f420974453b9b6b0fe17795378b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.297355 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fb42b54f558499090a1d7cf257dd84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.300671 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 209fa71af13142a58efb9f3dab3eecd3 (DEFAULT_TABLE table=default.Table_f4fc99282b5643e89641bb7082d165a5 [id=1e6385207c1648f8a87346a3ca603cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.301101 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 209fa71af13142a58efb9f3dab3eecd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.303411 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53f25688e3b45dca68984e2c543d094. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.304023 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfd3e23d83a84648bd11cd258377aecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.308395 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 9b9a5daa36b547a7888d38dc3329eaf2 (DEFAULT_TABLE table=default.Table_df797fe86da84688840a0e8a2d2f18ed [id=26a5eae9586f4d00b5ebaf9eb3462272]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.308748 1956 tablet_service.cc:1505] Processing CreateTablet for tablet e203fe0403594d30b44d531153d42a10 (DEFAULT_TABLE table=default.Table_874fdf4c2b4e4ace9b87c72de1e573a7 [id=4226c45f4f204b3ca19dd2e0c7a15fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.308943 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 6ea364c4a5cf4f3f99d9ff6904add697 (DEFAULT_TABLE table=default.Table_b0a34356c6e54ae79696bc5edfeb8aa1 [id=02c334c1e9e5480d8ac279c0b5e62800]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.309062 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41a1442fffe54add93666e3c78a0cc58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.309353 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ea364c4a5cf4f3f99d9ff6904add697. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.311625 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e203fe0403594d30b44d531153d42a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.313169 2372 tablet_replica.cc:333] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:28.313225 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a6f9ae327aa4401baf66a737eaa93cf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6f9ae327aa4401baf66a737eaa93cf6 already in progress: creating tablet
W20251026 02:56:28.313683 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a012d868a78f4324a92db83c81931b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a012d868a78f4324a92db83c81931b9d already in progress: creating tablet
I20251026 02:56:28.316056 2661 raft_consensus.cc:493] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:28.316236 2661 raft_consensus.cc:515] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.317443 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 85420a102dae4269811dbdc64aebb20a (DEFAULT_TABLE table=default.Table_559b75aad2cb44b196cc0ab1dbf2c550 [id=240bb6da01cf4c1b98f9301e17cc7817]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.317611 2661 leader_election.cc:290] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:28.317836 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85420a102dae4269811dbdc64aebb20a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:28.318423 1916 leader_election.cc:343] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: cfed3523205548508d1101586ee3e2a9
I20251026 02:56:28.318571 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9a5daa36b547a7888d38dc3329eaf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.319329 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 209fa71af13142a58efb9f3dab3eecd3 (DEFAULT_TABLE table=default.Table_f4fc99282b5643e89641bb7082d165a5 [id=1e6385207c1648f8a87346a3ca603cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.319774 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 209fa71af13142a58efb9f3dab3eecd3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:28.320595 1915 leader_election.cc:343] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: cfed3523205548508d1101586ee3e2a9
I20251026 02:56:28.320904 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e89a98fcfdbb4011876b0ebc2af91f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.321252 2372 raft_consensus.cc:2243] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:28.321527 2372 raft_consensus.cc:2272] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:28.322153 2372 ts_tablet_manager.cc:1916] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:28.296895 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78a624d839934d3089b64e397d4360af" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:28.324285 1916 leader_election.cc:304] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:28.326449 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e89a98fcfdbb4011876b0ebc2af91f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.328768 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 87895fb88c284a4f8d7c58a5ee48ba8e (DEFAULT_TABLE table=default.Table_000a2b7754d04bd197fca93eab72c154 [id=f28b682102ae461e96172ab3c0529193]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.329234 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87895fb88c284a4f8d7c58a5ee48ba8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.330041 1915 leader_election.cc:304] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:28.330138 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 85420a102dae4269811dbdc64aebb20a (DEFAULT_TABLE table=default.Table_559b75aad2cb44b196cc0ab1dbf2c550 [id=240bb6da01cf4c1b98f9301e17cc7817]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.330562 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85420a102dae4269811dbdc64aebb20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.331279 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 6ea364c4a5cf4f3f99d9ff6904add697 (DEFAULT_TABLE table=default.Table_b0a34356c6e54ae79696bc5edfeb8aa1 [id=02c334c1e9e5480d8ac279c0b5e62800]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.331667 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ea364c4a5cf4f3f99d9ff6904add697. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.333417 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 6699b50d87964f6a89c365171253ad25 (DEFAULT_TABLE table=default.Table_3792bcbbc16b4be0b113cb5f07b9261b [id=8a3d1c20763445c499bd655eb99deb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.335553 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6699b50d87964f6a89c365171253ad25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.335158 2087 tablet_service.cc:1505] Processing CreateTablet for tablet bc4ca4423e3a4e0c96846babec931fec (DEFAULT_TABLE table=default.Table_91f57fb05381498c89f99d19bdc7cd9b [id=05a30266e1a04e1490f58d1feb80b2a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.336277 2661 raft_consensus.cc:2749] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:28.336720 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc4ca4423e3a4e0c96846babec931fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.336865 1952 tablet_service.cc:1505] Processing CreateTablet for tablet a53f25688e3b45dca68984e2c543d094 (DEFAULT_TABLE table=default.Table_feb822201306444ebe478734d286a667 [id=535d3143b66249a083853a11934d994e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.340299 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 9fb42b54f558499090a1d7cf257dd84c (DEFAULT_TABLE table=default.Table_8fbaf8dac35d47848025cbb47c261c08 [id=9ac57a0fe65c4e07b113f83dacca8043]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.340343 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 209fa71af13142a58efb9f3dab3eecd3 (DEFAULT_TABLE table=default.Table_f4fc99282b5643e89641bb7082d165a5 [id=1e6385207c1648f8a87346a3ca603cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.340644 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53f25688e3b45dca68984e2c543d094. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.340562 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 87895fb88c284a4f8d7c58a5ee48ba8e (DEFAULT_TABLE table=default.Table_000a2b7754d04bd197fca93eab72c154 [id=f28b682102ae461e96172ab3c0529193]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.340951 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87895fb88c284a4f8d7c58a5ee48ba8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.340960 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fb42b54f558499090a1d7cf257dd84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.341217 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 209fa71af13142a58efb9f3dab3eecd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.342420 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 6699b50d87964f6a89c365171253ad25 (DEFAULT_TABLE table=default.Table_3792bcbbc16b4be0b113cb5f07b9261b [id=8a3d1c20763445c499bd655eb99deb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.342778 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6699b50d87964f6a89c365171253ad25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.346252 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f83c29969baa4d62bca3154cc3d2cc33" } new_table_name: "default.Table_464e60af830f4416af8fdcf84f660a49" modify_external_catalogs: true
I20251026 02:56:28.347680 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 122de8616b8f4fcf88971d5738c47e82
I20251026 02:56:28.348196 1661 catalog_manager.cc:6719] ReplaceTablet: tablet 122de8616b8f4fcf88971d5738c47e82 deleted and replaced by tablet 81775055d98349f696cdab6171ffd58b
I20251026 02:56:28.348639 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b3182aeed9424bcb9d78331e40f7867f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.349169 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3182aeed9424bcb9d78331e40f7867f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.351824 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 122de8616b8f4fcf88971d5738c47e82 with delete_type TABLET_DATA_DELETED (replaced by tablet 81775055d98349f696cdab6171ffd58b) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:28.352345 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 3b77bb6b4722449c9699b548a9ef6314 (DEFAULT_TABLE table=default.Table_34881d9847d5463297830bf6cc4ed1e6 [id=a484f36091f24fd2b8bc2af4f53eb450]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.352757 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b77bb6b4722449c9699b548a9ef6314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.354259 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 85420a102dae4269811dbdc64aebb20a (DEFAULT_TABLE table=default.Table_559b75aad2cb44b196cc0ab1dbf2c550 [id=240bb6da01cf4c1b98f9301e17cc7817]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.354737 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85420a102dae4269811dbdc64aebb20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.358055 2661 raft_consensus.cc:2749] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:28.359228 1949 tablet_service.cc:1505] Processing CreateTablet for tablet 87895fb88c284a4f8d7c58a5ee48ba8e (DEFAULT_TABLE table=default.Table_000a2b7754d04bd197fca93eab72c154 [id=f28b682102ae461e96172ab3c0529193]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.360541 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 17d2cc6b16d34331bb80c576c0cd2ce0
I20251026 02:56:28.361459 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c2ef795e86444f9f85bb7d0bcca51240" } modify_external_catalogs: true
I20251026 02:56:28.361652 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c2ef795e86444f9f85bb7d0bcca51240" } modify_external_catalogs: true
I20251026 02:56:28.363456 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a53f25688e3b45dca68984e2c543d094
I20251026 02:56:28.364197 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3afd36d31ad34be682d7a49ab29da62b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:28.364961 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6281fc8054c54d01b0376df4725d1adc" } new_table_name: "default.Table_52417f25dbdb4676a5aaaca9bf46962c" modify_external_catalogs: true
I20251026 02:56:28.365831 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 209fa71af13142a58efb9f3dab3eecd3
I20251026 02:56:28.365954 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet aa10cd990a944eab8d9eb66aaf20080c
I20251026 02:56:28.365319 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8611fe991ff441caab2cd1faa546ddea" } new_table_name: "default.Table_b709e5e9bb364edc939c1b743ca9009e" modify_external_catalogs: true
I20251026 02:56:28.372793 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet aa10cd990a944eab8d9eb66aaf20080c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:28.372973 2375 tablet_bootstrap.cc:492] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:28.373145 2375 ts_tablet_manager.cc:1403] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.252s user 0.007s sys 0.003s
I20251026 02:56:28.373914 2375 raft_consensus.cc:359] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.374141 2375 raft_consensus.cc:385] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.374236 2375 raft_consensus.cc:740] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.374467 2375 consensus_queue.cc:260] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.375121 2375 ts_tablet_manager.cc:1434] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:28.375533 2375 tablet_bootstrap.cc:492] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:28.375721 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3afd36d31ad34be682d7a49ab29da62b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.377415 1949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87895fb88c284a4f8d7c58a5ee48ba8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.378492 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 17d2cc6b16d34331bb80c576c0cd2ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:28.379278 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet a53f25688e3b45dca68984e2c543d094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:28.379602 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 209fa71af13142a58efb9f3dab3eecd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:28.383625 2194 consensus_peers.cc:597] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:28.387741 2372 ts_tablet_manager.cc:1929] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:28.387952 2372 log.cc:1199] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/8dbc12d49b094dbeadad2e76916ba9af
I20251026 02:56:28.388444 2372 ts_tablet_manager.cc:1950] T 8dbc12d49b094dbeadad2e76916ba9af P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:28.389250 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 8dbc12d49b094dbeadad2e76916ba9af (table default.Table_cdb10d198a394aac9033a98152fd2d94 [id=1d0784496733469fbff42783f90d56d2]) successfully deleted
I20251026 02:56:28.391091 2240 tablet_service.cc:1505] Processing CreateTablet for tablet bc4ca4423e3a4e0c96846babec931fec (DEFAULT_TABLE table=default.Table_91f57fb05381498c89f99d19bdc7cd9b [id=05a30266e1a04e1490f58d1feb80b2a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.391510 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc4ca4423e3a4e0c96846babec931fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.392484 2372 tablet_replica.cc:333] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:28.392768 2372 raft_consensus.cc:2243] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:28.393000 2372 raft_consensus.cc:2272] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:28.393704 2372 ts_tablet_manager.cc:1916] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:28.396976 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c4af4099c79b4cd9a7e7ef8a0d762dbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.397846 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4af4099c79b4cd9a7e7ef8a0d762dbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.399356 2899 tablet.cc:1722] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:28.399436 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 6ea364c4a5cf4f3f99d9ff6904add697 (DEFAULT_TABLE table=default.Table_b0a34356c6e54ae79696bc5edfeb8aa1 [id=02c334c1e9e5480d8ac279c0b5e62800]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.399618 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4098599e818f4dffbf00a6439bb66d9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:28.400485 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0cd100667df948d29f7f04a6f55b44e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.401103 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0cd100667df948d29f7f04a6f55b44e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.405939 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet 122de8616b8f4fcf88971d5738c47e82 with delete_type TABLET_DATA_DELETED (replaced by tablet 81775055d98349f696cdab6171ffd58b) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:28.406172 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4098599e818f4dffbf00a6439bb66d9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.411525 2375 tablet_bootstrap.cc:654] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.415125 2381 consensus_queue.cc:260] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.415406 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 03da415f3a944a9a9679b3a45295ce35 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:28.416198 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_af9006c41e484127a39c22ff0130d281" } new_table_name: "default.Table_164cb1784f4645e499339f7163fdfc2b" modify_external_catalogs: true
I20251026 02:56:28.417533 2911 tablet.cc:1722] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:28.418766 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet a53f25688e3b45dca68984e2c543d094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:28.422361 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 122de8616b8f4fcf88971d5738c47e82 with delete_type TABLET_DATA_DELETED (replaced by tablet 81775055d98349f696cdab6171ffd58b) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:28.423528 2661 raft_consensus.cc:493] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:28.423489 2238 tablet_service.cc:1505] Processing CreateTablet for tablet 3b77bb6b4722449c9699b548a9ef6314 (DEFAULT_TABLE table=default.Table_34881d9847d5463297830bf6cc4ed1e6 [id=a484f36091f24fd2b8bc2af4f53eb450]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.423936 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b77bb6b4722449c9699b548a9ef6314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.425904 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 17d2cc6b16d34331bb80c576c0cd2ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:28.427183 2381 ts_tablet_manager.cc:1434] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.268s user 0.003s sys 0.000s
I20251026 02:56:28.428180 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet 209fa71af13142a58efb9f3dab3eecd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:28.428328 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 17d2cc6b16d34331bb80c576c0cd2ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:28.428665 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet aa10cd990a944eab8d9eb66aaf20080c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:28.431102 2381 tablet_bootstrap.cc:492] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:28.433609 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e701d4023fee45d18c48997aab3227c2 consensus state has no leader
I20251026 02:56:28.431290 1947 tablet_service.cc:1505] Processing CreateTablet for tablet 6699b50d87964f6a89c365171253ad25 (DEFAULT_TABLE table=default.Table_3792bcbbc16b4be0b113cb5f07b9261b [id=8a3d1c20763445c499bd655eb99deb56]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:28.434751 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 882cd1680aa140f6a63a961cd4aec25a consensus state has no leader
I20251026 02:56:28.428182 1946 tablet_service.cc:1505] Processing CreateTablet for tablet bc4ca4423e3a4e0c96846babec931fec (DEFAULT_TABLE table=default.Table_91f57fb05381498c89f99d19bdc7cd9b [id=05a30266e1a04e1490f58d1feb80b2a7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:28.435179 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31b6e08597bd450eb1875201cf594a82 has no consensus state
I20251026 02:56:28.436313 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet efd751ee4b1d433198dcd00c081f648d
I20251026 02:56:28.434855 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 3b77bb6b4722449c9699b548a9ef6314 (DEFAULT_TABLE table=default.Table_34881d9847d5463297830bf6cc4ed1e6 [id=a484f36091f24fd2b8bc2af4f53eb450]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.440418 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet a53f25688e3b45dca68984e2c543d094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:28.439682 2661 raft_consensus.cc:515] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.444245 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 209fa71af13142a58efb9f3dab3eecd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:28.444705 1650 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet c4b0933f71cb4ac7a58986d16b613e64 (table default.Table_a59579db048a4eec8a0a47544626d25f [id=cba6d4517d244d7580e525729e871745], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c4b0933f71cb4ac7a58986d16b613e64 consensus state has no leader
I20251026 02:56:28.445428 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet efd751ee4b1d433198dcd00c081f648d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:28.452925 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5a00bc04b3654f90af342986751344e6 consensus state has no leader
I20251026 02:56:28.454802 2372 ts_tablet_manager.cc:1929] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:28.454948 2372 log.cc:1199] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/84ad6c72a8704c148411881760848df4
I20251026 02:56:28.455348 2372 ts_tablet_manager.cc:1950] T 84ad6c72a8704c148411881760848df4 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:28.455857 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 84ad6c72a8704c148411881760848df4 (table default.Table_7814874203fc40af9ef07eeb2e11ebc4 [id=cc22b2daa8fb4eb1bae52b189d66b8fb]) successfully deleted
W20251026 02:56:28.457363 1916 leader_election.cc:343] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 461412f3ad1e4186b1302cdf5a624ed4
I20251026 02:56:28.460399 2372 tablet_replica.cc:333] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:28.460669 2372 raft_consensus.cc:2243] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:28.460690 2661 leader_election.cc:290] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:28.460886 2372 raft_consensus.cc:2272] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:28.461181 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 66ede86c604e47dbb982e5a3ba18bcb7 (DEFAULT_TABLE table=default.Table_503013f4998040b8a8d436174cfef014 [id=4407a702280f4f07a83342cc62da1e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.461484 2372 ts_tablet_manager.cc:1916] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:28.462774 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 868dbbfce40c4dd6b36cc220966b1986 (DEFAULT_TABLE table=default.Table_a2e4dfe2f01f4d8fac1b89211f87d9a6 [id=6ed9ef4d13484e92a0b292ed4085cea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.463377 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66ede86c604e47dbb982e5a3ba18bcb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.463702 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 868dbbfce40c4dd6b36cc220966b1986. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.468753 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 3ac38e6257e74367b06b15fc49d2af81 (DEFAULT_TABLE table=default.Table_ff6f3f84ac2348b79403d10fc9e3ecb5 [id=81ecbd0df65d45e4b8f1af69ca4fd488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.469259 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ac38e6257e74367b06b15fc49d2af81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.470567 2381 tablet_bootstrap.cc:654] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.472357 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 7834a85168364390a841fc883681c845 (DEFAULT_TABLE table=default.Table_ecfe7bd401bb42f2bcdcbb956e55779d [id=acf34beac75d4537a2af3b818886880c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.472499 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 109073eb664d4090b7a2b786146e13fa (DEFAULT_TABLE table=default.Table_9f48bbcff2cc4a47b64cb3862caccc33 [id=da72d631aee04d10a6832b04f208587f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.472764 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7834a85168364390a841fc883681c845. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.472996 2091 tablet_service.cc:1505] Processing CreateTablet for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 (DEFAULT_TABLE table=default.Table_4293bc067ecf4838a1557178f2c53ec1 [id=3e78c92f28694cc1b4660f5f5000843a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.473361 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:28.473845 2194 consensus_peers.cc:597] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:28.476399 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 109073eb664d4090b7a2b786146e13fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.487109 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 885c10070d6942f7abe52c4558d887e6 (DEFAULT_TABLE table=default.Table_a37eaaa2d8244f6f95a762d240e1dffb [id=39a1785e03fd45b7a8ea0f14bf5a362a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.487579 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 885c10070d6942f7abe52c4558d887e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.483505 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet aa10cd990a944eab8d9eb66aaf20080c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:28.483047 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ea364c4a5cf4f3f99d9ff6904add697. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.497145 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 31b6e08597bd450eb1875201cf594a82 (DEFAULT_TABLE table=default.Table_b709e5e9bb364edc939c1b743ca9009e [id=a557708372bd4249969f80adf8641556]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.497556 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31b6e08597bd450eb1875201cf594a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.500016 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6699b50d87964f6a89c365171253ad25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.500111 1655 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_5f363b68bc2c4e7685f25e4eefa532a8 [id=43cb7f538f824ae38059645230cc7e94] alter complete (version 3)
I20251026 02:56:28.501019 2375 tablet_bootstrap.cc:492] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:28.501170 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc4ca4423e3a4e0c96846babec931fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.501222 2375 ts_tablet_manager.cc:1403] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.126s user 0.002s sys 0.002s
I20251026 02:56:28.502108 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b77bb6b4722449c9699b548a9ef6314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.502140 2375 raft_consensus.cc:359] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.502439 2375 raft_consensus.cc:385] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.502543 2375 raft_consensus.cc:740] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.505666 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 81775055d98349f696cdab6171ffd58b (DEFAULT_TABLE table=default.Table_719b11bde46041c5a20750db14381807 [id=4912ce6ea2304b5a9c21dd91a7cbd56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.506127 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81775055d98349f696cdab6171ffd58b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.506892 2897 raft_consensus.cc:493] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:28.507126 2381 tablet_bootstrap.cc:492] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:28.507071 2897 raft_consensus.cc:515] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.507300 2381 ts_tablet_manager.cc:1403] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.000s
I20251026 02:56:28.507280 1655 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_1361fbf89aa04321a7876c1e43e27089 [id=c162ef89d2f14e91a24a6d8c2c37c261] alter complete (version 4)
I20251026 02:56:28.507732 2897 leader_election.cc:290] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:28.508085 2381 raft_consensus.cc:359] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.508474 2381 raft_consensus.cc:385] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.508575 2381 raft_consensus.cc:740] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.509027 2381 consensus_queue.cc:260] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.509550 2381 ts_tablet_manager.cc:1434] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:28.511709 2897 raft_consensus.cc:493] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:28.511881 2897 raft_consensus.cc:515] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.512665 2897 leader_election.cc:290] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:28.513885 2381 tablet_bootstrap.cc:492] T 47bfff40c181432896d6898ea50cd025 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:28.520717 1655 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77] alter complete (version 1)
I20251026 02:56:28.524320 2381 tablet_bootstrap.cc:654] T 47bfff40c181432896d6898ea50cd025 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.525177 2375 consensus_queue.cc:260] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.526147 2375 ts_tablet_manager.cc:1434] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20251026 02:56:28.526525 2375 tablet_bootstrap.cc:492] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:28.531287 2372 ts_tablet_manager.cc:1929] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:28.531486 2372 log.cc:1199] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/f8533f1c191042ee9f2c0b99972fe931
I20251026 02:56:28.531905 2372 ts_tablet_manager.cc:1950] T f8533f1c191042ee9f2c0b99972fe931 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:28.532501 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet f8533f1c191042ee9f2c0b99972fe931 (table default.Table_e70cf6d4e0dc4017b484753f4b0250d9 [id=f4fb24ed75a24d81ae8f90619ded2ca9]) successfully deleted
I20251026 02:56:28.532859 2372 tablet_replica.cc:333] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:28.533067 2372 raft_consensus.cc:2243] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:28.533263 2372 raft_consensus.cc:2272] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:28.533841 2372 ts_tablet_manager.cc:1916] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:28.535547 2378 tablet_bootstrap.cc:492] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:28.536111 1981 raft_consensus.cc:3060] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:28.536546 2375 tablet_bootstrap.cc:654] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.542088 2378 ts_tablet_manager.cc:1403] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.344s user 0.016s sys 0.037s
I20251026 02:56:28.545117 2375 tablet_bootstrap.cc:492] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:28.545301 2375 ts_tablet_manager.cc:1403] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251026 02:56:28.545938 2372 ts_tablet_manager.cc:1929] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:28.546063 2372 log.cc:1199] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/9156314f98ac4ed9981f6f13fc585043
I20251026 02:56:28.546169 2375 raft_consensus.cc:359] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.546371 2375 raft_consensus.cc:385] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.546473 2375 raft_consensus.cc:740] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.546520 2372 ts_tablet_manager.cc:1950] T 9156314f98ac4ed9981f6f13fc585043 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:28.547118 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 9156314f98ac4ed9981f6f13fc585043 (table default.Table_4578a4ce81ea42cd878859a8167babff [id=f914c9974ac44d3f84c3d312649fe255]) successfully deleted
I20251026 02:56:28.547195 2372 tablet_replica.cc:333] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:28.547420 2372 raft_consensus.cc:2243] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:28.547583 2372 raft_consensus.cc:2272] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:28.547616 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 8c1e63ac8c5645f3ba1578792c0a0dae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c1e63ac8c5645f3ba1578792c0a0dae already in progress: creating tablet
I20251026 02:56:28.548254 2372 ts_tablet_manager.cc:1916] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:28.549011 2381 tablet_bootstrap.cc:492] T 47bfff40c181432896d6898ea50cd025 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:28.549216 2381 ts_tablet_manager.cc:1403] T 47bfff40c181432896d6898ea50cd025 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.001s
I20251026 02:56:28.550076 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 892a5fab2af1432eb31d92d9859ea2a0 (DEFAULT_TABLE table=default.Table_8c3d1109382545ffa157d6a3cb39f322 [id=a32ea6f43bb54db296a99aa3bdaa300f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.550088 2381 raft_consensus.cc:359] T 47bfff40c181432896d6898ea50cd025 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.550290 2381 raft_consensus.cc:385] T 47bfff40c181432896d6898ea50cd025 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.550390 2381 raft_consensus.cc:740] T 47bfff40c181432896d6898ea50cd025 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.550493 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 892a5fab2af1432eb31d92d9859ea2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.550659 2381 consensus_queue.cc:260] T 47bfff40c181432896d6898ea50cd025 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.551262 2381 ts_tablet_manager.cc:1434] T 47bfff40c181432896d6898ea50cd025 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:28.553964 2027 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:28.554673 1979 raft_consensus.cc:3060] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:28.556784 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 24f16e40a30c41788a6f9813c235c5ab (DEFAULT_TABLE table=default.Table_de67223546af408ead7b2d0d3979d78d [id=8f0f4568c97449e981049fb6ae579d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.557263 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f16e40a30c41788a6f9813c235c5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.557358 2894 raft_consensus.cc:493] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:28.557507 2894 raft_consensus.cc:515] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.558161 2894 leader_election.cc:290] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:28.560572 2091 tablet_service.cc:1505] Processing CreateTablet for tablet a3af2695c41a4439bc4092b7cb1b2edf (DEFAULT_TABLE table=default.Table_60d87251cafc4e6ca68c5e0affce8260 [id=830f99de0af8447080b4cf7f1265d80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.560926 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3af2695c41a4439bc4092b7cb1b2edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.561877 2381 tablet_bootstrap.cc:492] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:28.564424 2093 tablet_service.cc:1505] Processing CreateTablet for tablet db94738da3554fbda64c23fde3ef6c1b (DEFAULT_TABLE table=default.Table_4098599e818f4dffbf00a6439bb66d9a [id=928effe263004633a026f61a978a1e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.564704 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 3aea01a248b345cf81d4d9862744eb78 (DEFAULT_TABLE table=default.Table_b3182aeed9424bcb9d78331e40f7867f [id=74d6732406a448e49911361bdc64417a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.565251 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 24449cb9121642009dec41220e27b193 (DEFAULT_TABLE table=default.Table_c4af4099c79b4cd9a7e7ef8a0d762dbf [id=e8ca26900f06405d904f48d7387baf16]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.566362 2084 tablet_service.cc:1505] Processing CreateTablet for tablet b8fe7825fa8d46669772cf84bc4b3afc (DEFAULT_TABLE table=default.Table_c1abfb93d24a4f2a97d8d87a06c34ea5 [id=649f153da3b14f0b99acef230e010a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.567034 2375 consensus_queue.cc:260] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.567732 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db94738da3554fbda64c23fde3ef6c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.567816 2375 ts_tablet_manager.cc:1434] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.022s user 0.002s sys 0.001s
I20251026 02:56:28.568316 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8fe7825fa8d46669772cf84bc4b3afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.568802 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24449cb9121642009dec41220e27b193. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.569263 2098 tablet_service.cc:1505] Processing CreateTablet for tablet c966cae6886048a2b6ad3b8bd7007bcb (DEFAULT_TABLE table=default.Table_3afd36d31ad34be682d7a49ab29da62b [id=f7982d61f7d8411193899caaa357b61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.569422 2381 tablet_bootstrap.cc:654] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.569749 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aea01a248b345cf81d4d9862744eb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.568150 2375 tablet_bootstrap.cc:492] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:28.573686 2119 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.007s sys 0.016s
W20251026 02:56:28.573807 2119 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.508s user 0.007s sys 0.016s
I20251026 02:56:28.574712 2897 raft_consensus.cc:493] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:28.574853 2897 raft_consensus.cc:515] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.575837 2897 leader_election.cc:290] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:28.582626 2381 tablet_bootstrap.cc:492] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:28.583678 2381 ts_tablet_manager.cc:1403] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.002s
I20251026 02:56:28.584589 2381 raft_consensus.cc:359] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.584865 2381 raft_consensus.cc:385] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.584956 2381 raft_consensus.cc:740] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.589258 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c966cae6886048a2b6ad3b8bd7007bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.586154 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 79c18ebf56e045b6a52fae8871a9107f (DEFAULT_TABLE table=default.Table_0cd100667df948d29f7f04a6f55b44e5 [id=1226fd50a18e4de1b12e9dda2e89b880]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.585314 2381 consensus_queue.cc:260] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.594439 2381 ts_tablet_manager.cc:1434] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251026 02:56:28.594442 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47bfff40c181432896d6898ea50cd025" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:28.594568 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1370b9ad917344c093e5da3d0e9b1cf0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:28.594738 2260 raft_consensus.cc:2468] T 47bfff40c181432896d6898ea50cd025 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:28.594862 2381 tablet_bootstrap.cc:492] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:28.596701 2381 tablet_bootstrap.cc:654] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.602262 2308 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:28.606679 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79c18ebf56e045b6a52fae8871a9107f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.607975 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (table default.Table_c9c7b9787c3a458ea41be132b0841599 [id=1d09f5b9fa164ac783f0be9332eac4c3]). Expected version 5 got 0
I20251026 02:56:28.610226 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet a821ed8a686c452a975acb96b5c74778 (table default.Table_00dc4c49a19246889353df2157eebf15 [id=347d712aca4e477c89805846800fb824]). Expected version 1 got 0
W20251026 02:56:28.612205 2847 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 2330091848794891954) tablet 00000000000000000000000000000000 cur holder 0x617000051680
I20251026 02:56:28.613147 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet c570e5f9a93c45bda32ec92c68794714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:28.621670 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet a70d70abc3a34dac8cf9a747903fa78c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:28.625669 2381 tablet_bootstrap.cc:492] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:28.625867 2381 ts_tablet_manager.cc:1403] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.007s
I20251026 02:56:28.626917 2381 raft_consensus.cc:359] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.627159 2381 raft_consensus.cc:385] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.627265 2381 raft_consensus.cc:740] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.627589 2381 consensus_queue.cc:260] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.629969 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (table default.Table_c9c7b9787c3a458ea41be132b0841599 [id=1d09f5b9fa164ac783f0be9332eac4c3]). Expected version 5 got 4
I20251026 02:56:28.634332 2381 ts_tablet_manager.cc:1434] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20251026 02:56:28.639576 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a70d70abc3a34dac8cf9a747903fa78c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:28.641515 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet efd751ee4b1d433198dcd00c081f648d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:28.641522 2053 leader_election.cc:343] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:28.641848 2053 leader_election.cc:304] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:28.642083 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet e62064fbdffb4954a3a1227efd6327ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:28.642321 2195 leader_election.cc:343] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 84ad6c72a8704c148411881760848df4
I20251026 02:56:28.643047 2381 tablet_bootstrap.cc:492] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:28.648826 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6e720c10b85f4046a9617986f013c678" } new_table_name: "default.Table_9fb0008346da44318ccc90cd69474750" modify_external_catalogs: true
I20251026 02:56:28.648882 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_09b68d3ebab94c55b3c90567af20db3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:28.648940 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_292ad60362294da0976fd8fe6b180c0a" } modify_external_catalogs: true
I20251026 02:56:28.649276 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_292ad60362294da0976fd8fe6b180c0a" } modify_external_catalogs: true
I20251026 02:56:28.649390 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8f3c2b96b146407f87972ce3efb7e27d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.649653 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09b68d3ebab94c55b3c90567af20db3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:28.649931 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f3c2b96b146407f87972ce3efb7e27d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.651646 2897 raft_consensus.cc:2804] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:28.651708 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4bd0ad4431c84593ae8581d943392a9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.652318 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bd0ad4431c84593ae8581d943392a9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.653633 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_51c1281363e94fcaa024e7e1aca9b333"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.654155 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51c1281363e94fcaa024e7e1aca9b333 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.655364 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5ae1979aa25849fe8b780ecc617bb7e8" } modify_external_catalogs: true
I20251026 02:56:28.655555 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5ae1979aa25849fe8b780ecc617bb7e8" } modify_external_catalogs: true
W20251026 02:56:28.655886 2053 leader_election.cc:343] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:28.656857 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f4bddfca37d64369b99a1fdc0102ca35" } modify_external_catalogs: true
I20251026 02:56:28.657019 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f4bddfca37d64369b99a1fdc0102ca35" } modify_external_catalogs: true
I20251026 02:56:28.657089 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ce7dea3ace314a5fa3ffeb9f16512725" } modify_external_catalogs: true
I20251026 02:56:28.657243 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ce7dea3ace314a5fa3ffeb9f16512725" } modify_external_catalogs: true
I20251026 02:56:28.659008 2897 raft_consensus.cc:493] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:28.659154 2897 raft_consensus.cc:3060] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:28.666781 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 41b7b0c36ba748f0a1565892dbe98a78
I20251026 02:56:28.667050 2892 raft_consensus.cc:493] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:28.670888 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 868dbbfce40c4dd6b36cc220966b1986 (DEFAULT_TABLE table=default.Table_a2e4dfe2f01f4d8fac1b89211f87d9a6 [id=6ed9ef4d13484e92a0b292ed4085cea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.671408 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 868dbbfce40c4dd6b36cc220966b1986. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.672852 2894 raft_consensus.cc:493] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:28.675631 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 66ede86c604e47dbb982e5a3ba18bcb7 (DEFAULT_TABLE table=default.Table_503013f4998040b8a8d436174cfef014 [id=4407a702280f4f07a83342cc62da1e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.676342 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47bfff40c181432896d6898ea50cd025" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:28.676872 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1370b9ad917344c093e5da3d0e9b1cf0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:28.681687 2375 tablet_bootstrap.cc:654] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.683897 2892 raft_consensus.cc:515] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.684932 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 41b7b0c36ba748f0a1565892dbe98a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:28.685362 2897 raft_consensus.cc:515] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.686514 2897 leader_election.cc:290] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:28.687546 2238 tablet_service.cc:1505] Processing CreateTablet for tablet 109073eb664d4090b7a2b786146e13fa (DEFAULT_TABLE table=default.Table_9f48bbcff2cc4a47b64cb3862caccc33 [id=da72d631aee04d10a6832b04f208587f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.686663 2894 raft_consensus.cc:515] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.689088 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 7834a85168364390a841fc883681c845 (DEFAULT_TABLE table=default.Table_ecfe7bd401bb42f2bcdcbb956e55779d [id=acf34beac75d4537a2af3b818886880c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.691166 2894 leader_election.cc:290] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:28.694381 2631 raft_consensus.cc:493] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:28.694875 2240 tablet_service.cc:1505] Processing CreateTablet for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 (DEFAULT_TABLE table=default.Table_4293bc067ecf4838a1557178f2c53ec1 [id=3e78c92f28694cc1b4660f5f5000843a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.695104 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66ede86c604e47dbb982e5a3ba18bcb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.697608 2911 tablet.cc:1722] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:28.698767 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 31b6e08597bd450eb1875201cf594a82 (DEFAULT_TABLE table=default.Table_b709e5e9bb364edc939c1b743ca9009e [id=a557708372bd4249969f80adf8641556]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.704494 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 109073eb664d4090b7a2b786146e13fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.706846 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 3ac38e6257e74367b06b15fc49d2af81 (DEFAULT_TABLE table=default.Table_ff6f3f84ac2348b79403d10fc9e3ecb5 [id=81ecbd0df65d45e4b8f1af69ca4fd488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.715292 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 84ad6c72a8704c148411881760848df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:28.715564 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 892a5fab2af1432eb31d92d9859ea2a0 (DEFAULT_TABLE table=default.Table_8c3d1109382545ffa157d6a3cb39f322 [id=a32ea6f43bb54db296a99aa3bdaa300f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.719740 2372 ts_tablet_manager.cc:1929] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251026 02:56:28.719933 2053 leader_election.cc:343] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:28.721396 2053 leader_election.cc:343] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:28.721573 2053 leader_election.cc:304] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:28.715500 2892 leader_election.cc:290] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:28.733726 2897 raft_consensus.cc:2749] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:28.736850 2308 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:28.742022 2372 log.cc:1199] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7845ebd3e9ca4897b8fb19a48a010315
I20251026 02:56:28.742389 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 885c10070d6942f7abe52c4558d887e6 (DEFAULT_TABLE table=default.Table_a37eaaa2d8244f6f95a762d240e1dffb [id=39a1785e03fd45b7a8ea0f14bf5a362a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.742606 2372 ts_tablet_manager.cc:1950] T 7845ebd3e9ca4897b8fb19a48a010315 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:28.742790 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet aa399e6407fd4c4d9871d3d5ccf894ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:28.743696 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 353) took 1192 ms. Trace:
I20251026 02:56:28.743954 2372 rpcz_store.cc:276] 1026 02:56:27.551077 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.551938 (+ 861us) service_pool.cc:225] Handling call
1026 02:56:28.547139 (+995201us) ts_tablet_manager.cc:1151] Deleting tablet 7845ebd3e9ca4897b8fb19a48a010315
1026 02:56:28.547146 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:28.646998 (+ 99852us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.719558 (+ 72560us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.743629 (+ 24071us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":995019}
I20251026 02:56:28.745841 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7845ebd3e9ca4897b8fb19a48a010315 (table default.Table_9c97bccbf4cb4d15a66dbb8130acb6d4 [id=10d722da66cd4661a1bb1ab82997dca8]) successfully deleted
I20251026 02:56:28.747049 2165 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:28.747287 2165 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:28.751212 2381 tablet_bootstrap.cc:654] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.751147 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 24f16e40a30c41788a6f9813c235c5ab (DEFAULT_TABLE table=default.Table_de67223546af408ead7b2d0d3979d78d [id=8f0f4568c97449e981049fb6ae579d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.752269 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31b6e08597bd450eb1875201cf594a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.754419 2243 tablet_service.cc:1505] Processing CreateTablet for tablet b8fe7825fa8d46669772cf84bc4b3afc (DEFAULT_TABLE table=default.Table_c1abfb93d24a4f2a97d8d87a06c34ea5 [id=649f153da3b14f0b99acef230e010a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.754760 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7834a85168364390a841fc883681c845. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.754948 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 357) took 1190 ms. Trace:
I20251026 02:56:28.755000 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.755056 2372 rpcz_store.cc:276] 1026 02:56:27.564490 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.564560 (+ 70us) service_pool.cc:225] Handling call
1026 02:56:28.746264 (+1181704us) ts_tablet_manager.cc:1151] Deleting tablet cfc3aed64ad0439cb22fd6815c973d87
1026 02:56:28.746280 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:28.754887 (+ 8607us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1181366}
W20251026 02:56:28.755781 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet cfc3aed64ad0439cb22fd6815c973d87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfc3aed64ad0439cb22fd6815c973d87 already in progress: creating tablet
I20251026 02:56:28.753322 2631 raft_consensus.cc:515] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.756858 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 24449cb9121642009dec41220e27b193 (DEFAULT_TABLE table=default.Table_c4af4099c79b4cd9a7e7ef8a0d762dbf [id=e8ca26900f06405d904f48d7387baf16]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.760380 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ac38e6257e74367b06b15fc49d2af81. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:28.749423 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 363482us
I20251026 02:56:28.761693 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 358) took 1186 ms. Trace:
I20251026 02:56:28.761809 2372 rpcz_store.cc:276] 1026 02:56:27.575252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.575311 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:28.756170 (+1180859us) ts_tablet_manager.cc:1151] Deleting tablet d43b6f22dd4f4f9c9657f21514550312
1026 02:56:28.761483 (+ 5313us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:28.761627 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1180541}
W20251026 02:56:28.762409 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet d43b6f22dd4f4f9c9657f21514550312 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43b6f22dd4f4f9c9657f21514550312 already in progress: creating tablet
I20251026 02:56:28.762899 2372 tablet_replica.cc:333] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:28.763098 2239 tablet_service.cc:1505] Processing CreateTablet for tablet db94738da3554fbda64c23fde3ef6c1b (DEFAULT_TABLE table=default.Table_4098599e818f4dffbf00a6439bb66d9a [id=928effe263004633a026f61a978a1e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.763170 2372 raft_consensus.cc:2243] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:28.763442 2372 raft_consensus.cc:2272] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:28.763522 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 81775055d98349f696cdab6171ffd58b (DEFAULT_TABLE table=default.Table_719b11bde46041c5a20750db14381807 [id=4912ce6ea2304b5a9c21dd91a7cbd56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.764037 2372 ts_tablet_manager.cc:1916] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:28.766700 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47bfff40c181432896d6898ea50cd025" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:28.767478 2115 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22e4a433801a4cb58062d3e76c760b69" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:28.767709 2115 raft_consensus.cc:2468] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:28.768380 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4011930a277d410f863a1558416c4b0f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:28.768586 2118 raft_consensus.cc:2468] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:28.769490 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa399e6407fd4c4d9871d3d5ccf894ec" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:28.769927 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47bfff40c181432896d6898ea50cd025" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:28.775277 2899 tablet.cc:1722] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:28.775799 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f16e40a30c41788a6f9813c235c5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.776333 2631 leader_election.cc:290] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:28.788455 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet a821ed8a686c452a975acb96b5c74778 (table default.Table_00dc4c49a19246889353df2157eebf15 [id=347d712aca4e477c89805846800fb824]). Expected version 1 got 0
I20251026 02:56:28.788612 2375 tablet_bootstrap.cc:492] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:28.789337 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b120712c89884acc8703a30980f5a265"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:28.789544 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_cf205f15299443d6a95fd58c1a38ee25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.789999 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b120712c89884acc8703a30980f5a265 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.790400 2375 ts_tablet_manager.cc:1403] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.222s user 0.022s sys 0.014s
I20251026 02:56:28.791487 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fafb3f2678b846e3bf05a461dde63011" } new_table_name: "default.Table_10b9ad21b01e4323a1bdb23caf300803" modify_external_catalogs: true
I20251026 02:56:28.789358 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8fbaf8dac35d47848025cbb47c261c08" } new_table_name: "default.Table_70dab64d9fe341f987afbff383e463d9" modify_external_catalogs: true
W20251026 02:56:28.789999 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf205f15299443d6a95fd58c1a38ee25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.793869 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet efd751ee4b1d433198dcd00c081f648d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:28.793838 2927 tablet.cc:1722] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251026 02:56:28.796375 1915 leader_election.cc:343] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 461412f3ad1e4186b1302cdf5a624ed4
I20251026 02:56:28.796741 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 66ede86c604e47dbb982e5a3ba18bcb7 (DEFAULT_TABLE table=default.Table_503013f4998040b8a8d436174cfef014 [id=4407a702280f4f07a83342cc62da1e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.801501 2260 raft_consensus.cc:3060] T 47bfff40c181432896d6898ea50cd025 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:28.801590 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 868dbbfce40c4dd6b36cc220966b1986 (DEFAULT_TABLE table=default.Table_a2e4dfe2f01f4d8fac1b89211f87d9a6 [id=6ed9ef4d13484e92a0b292ed4085cea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.802788 2195 leader_election.cc:304] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:28.785111 2227 tablet_service.cc:1505] Processing CreateTablet for tablet 3aea01a248b345cf81d4d9862744eb78 (DEFAULT_TABLE table=default.Table_b3182aeed9424bcb9d78331e40f7867f [id=74d6732406a448e49911361bdc64417a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.803627 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a70d70abc3a34dac8cf9a747903fa78c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:28.804322 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 892a5fab2af1432eb31d92d9859ea2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.806136 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81775055d98349f696cdab6171ffd58b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.805548 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 79c18ebf56e045b6a52fae8871a9107f (DEFAULT_TABLE table=default.Table_0cd100667df948d29f7f04a6f55b44e5 [id=1226fd50a18e4de1b12e9dda2e89b880]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.807181 2378 raft_consensus.cc:359] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.807390 2378 raft_consensus.cc:385] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.808653 2226 tablet_service.cc:1505] Processing CreateTablet for tablet c966cae6886048a2b6ad3b8bd7007bcb (DEFAULT_TABLE table=default.Table_3afd36d31ad34be682d7a49ab29da62b [id=f7982d61f7d8411193899caaa357b61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.805642 2228 tablet_service.cc:1505] Processing CreateTablet for tablet a3af2695c41a4439bc4092b7cb1b2edf (DEFAULT_TABLE table=default.Table_60d87251cafc4e6ca68c5e0affce8260 [id=830f99de0af8447080b4cf7f1265d80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.810259 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 3ac38e6257e74367b06b15fc49d2af81 (DEFAULT_TABLE table=default.Table_ff6f3f84ac2348b79403d10fc9e3ecb5 [id=81ecbd0df65d45e4b8f1af69ca4fd488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.810652 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet 41b7b0c36ba748f0a1565892dbe98a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:28.810946 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db94738da3554fbda64c23fde3ef6c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.808317 1915 leader_election.cc:304] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:28.815493 1949 tablet_service.cc:1505] Processing CreateTablet for tablet 109073eb664d4090b7a2b786146e13fa (DEFAULT_TABLE table=default.Table_9f48bbcff2cc4a47b64cb3862caccc33 [id=da72d631aee04d10a6832b04f208587f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.824038 2377 ts_tablet_manager.cc:1929] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:28.824611 2631 raft_consensus.cc:2804] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:28.824657 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8fe7825fa8d46669772cf84bc4b3afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.824748 2631 raft_consensus.cc:493] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:28.824874 2631 raft_consensus.cc:3060] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:28.828305 2377 log.cc:1199] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/155a23a93358487a9564bb5ac175b4eb
I20251026 02:56:28.828644 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3af2695c41a4439bc4092b7cb1b2edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.828862 2377 ts_tablet_manager.cc:1950] T 155a23a93358487a9564bb5ac175b4eb P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:28.829391 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 312) took 2102 ms. Trace:
I20251026 02:56:28.829524 2377 rpcz_store.cc:276] 1026 02:56:26.727228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.727277 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:28.169158 (+1441881us) ts_tablet_manager.cc:1151] Deleting tablet 155a23a93358487a9564bb5ac175b4eb
1026 02:56:28.169170 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:28.668338 (+499168us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.808016 (+139678us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.829314 (+ 21298us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1441652}
I20251026 02:56:28.830145 2377 tablet_replica.cc:333] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:28.830427 2377 raft_consensus.cc:2243] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:28.830653 2377 raft_consensus.cc:2272] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:28.831406 2377 ts_tablet_manager.cc:1916] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:28.832533 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aea01a248b345cf81d4d9862744eb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.833436 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24449cb9121642009dec41220e27b193. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:28.838506 2195 consensus_peers.cc:597] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81530bbf4a2d4e908add683475e594fb. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:28.840169 2440 raft_consensus.cc:493] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:28.840565 2378 raft_consensus.cc:740] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.828109 2631 raft_consensus.cc:515] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.849341 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 81775055d98349f696cdab6171ffd58b (DEFAULT_TABLE table=default.Table_719b11bde46041c5a20750db14381807 [id=4912ce6ea2304b5a9c21dd91a7cbd56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.851557 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84ad6c72a8704c148411881760848df4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:28.852471 2378 consensus_queue.cc:260] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.854768 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22e4a433801a4cb58062d3e76c760b69" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:28.854966 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 44387.8us
I20251026 02:56:28.856542 1632 master_runner.cc:186] Error getting master registration for 127.1.150.60:33729: OK, instance_id { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" instance_seqno: 1761447385850427 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251026 02:56:28.856809 1632 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 02:56:28.857439 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 41b7b0c36ba748f0a1565892dbe98a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:28.857363 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d197f56404544c8cb996d53edb9559a4" } modify_external_catalogs: true
I20251026 02:56:28.857622 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d197f56404544c8cb996d53edb9559a4" } modify_external_catalogs: true
I20251026 02:56:28.857652 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 892a5fab2af1432eb31d92d9859ea2a0 (DEFAULT_TABLE table=default.Table_8c3d1109382545ffa157d6a3cb39f322 [id=a32ea6f43bb54db296a99aa3bdaa300f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.860222 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_852e528c9c8c460d82d9e34569c8860d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:28.860754 1946 tablet_service.cc:1505] Processing CreateTablet for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 (DEFAULT_TABLE table=default.Table_4293bc067ecf4838a1557178f2c53ec1 [id=3e78c92f28694cc1b4660f5f5000843a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:28.860939 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_852e528c9c8c460d82d9e34569c8860d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.861052 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 31b6e08597bd450eb1875201cf594a82 (DEFAULT_TABLE table=default.Table_b709e5e9bb364edc939c1b743ca9009e [id=a557708372bd4249969f80adf8641556]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.861307 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 7834a85168364390a841fc883681c845 (DEFAULT_TABLE table=default.Table_ecfe7bd401bb42f2bcdcbb956e55779d [id=acf34beac75d4537a2af3b818886880c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.862208 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3b7d85a2941545ceaa16b22e8328b4ff
I20251026 02:56:28.862138 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 885c10070d6942f7abe52c4558d887e6 (DEFAULT_TABLE table=default.Table_a37eaaa2d8244f6f95a762d240e1dffb [id=39a1785e03fd45b7a8ea0f14bf5a362a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.862376 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 24f16e40a30c41788a6f9813c235c5ab (DEFAULT_TABLE table=default.Table_de67223546af408ead7b2d0d3979d78d [id=8f0f4568c97449e981049fb6ae579d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.862583 1957 tablet_service.cc:1505] Processing CreateTablet for tablet a3af2695c41a4439bc4092b7cb1b2edf (DEFAULT_TABLE table=default.Table_60d87251cafc4e6ca68c5e0affce8260 [id=830f99de0af8447080b4cf7f1265d80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.862788 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 24449cb9121642009dec41220e27b193 (DEFAULT_TABLE table=default.Table_c4af4099c79b4cd9a7e7ef8a0d762dbf [id=e8ca26900f06405d904f48d7387baf16]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.862933 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6e87e5c48f4a4aa8be0b0ecc837e20dd" } new_table_name: "default.Table_7c68f14075ce4e58bc9d8f08191f3f70" modify_external_catalogs: true
I20251026 02:56:28.862993 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 79c18ebf56e045b6a52fae8871a9107f (DEFAULT_TABLE table=default.Table_0cd100667df948d29f7f04a6f55b44e5 [id=1226fd50a18e4de1b12e9dda2e89b880]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.863204 1977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4011930a277d410f863a1558416c4b0f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:28.863710 1943 tablet_service.cc:1505] Processing CreateTablet for tablet c966cae6886048a2b6ad3b8bd7007bcb (DEFAULT_TABLE table=default.Table_3afd36d31ad34be682d7a49ab29da62b [id=f7982d61f7d8411193899caaa357b61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.863991 1950 tablet_service.cc:1505] Processing CreateTablet for tablet db94738da3554fbda64c23fde3ef6c1b (DEFAULT_TABLE table=default.Table_4098599e818f4dffbf00a6439bb66d9a [id=928effe263004633a026f61a978a1e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.864017 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa399e6407fd4c4d9871d3d5ccf894ec" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:28.864347 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet dfd3e23d83a84648bd11cd258377aecb
I20251026 02:56:28.865038 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_173c6dd0f5b241c8a5ce1e918bde90a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.865554 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_173c6dd0f5b241c8a5ce1e918bde90a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:28.869004 2053 leader_election.cc:343] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:28.869068 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 3aea01a248b345cf81d4d9862744eb78 (DEFAULT_TABLE table=default.Table_b3182aeed9424bcb9d78331e40f7867f [id=74d6732406a448e49911361bdc64417a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:28.869362 2053 leader_election.cc:343] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:28.869553 2053 leader_election.cc:304] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:28.870118 2897 raft_consensus.cc:2749] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:28.873471 2661 raft_consensus.cc:2749] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:28.873791 2376 ts_tablet_manager.cc:1929] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:28.874837 2376 log.cc:1199] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/8667ffa25f2b478a987826d486400bbe
I20251026 02:56:28.875826 2376 ts_tablet_manager.cc:1950] T 8667ffa25f2b478a987826d486400bbe P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:28.878019 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (table default.Table_c9c7b9787c3a458ea41be132b0841599 [id=1d09f5b9fa164ac783f0be9332eac4c3]). Expected version 5 got 4
I20251026 02:56:28.878583 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet dbfe80d388644d0488cb2431a334c908
I20251026 02:56:28.879562 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7ba0c9a1ed6e49f09387b3e13faee903"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.880350 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ba0c9a1ed6e49f09387b3e13faee903 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.882455 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3f562a6edad942ad94cb473dddea9cd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.882961 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f562a6edad942ad94cb473dddea9cd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.884256 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet dfd3e23d83a84648bd11cd258377aecb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:28.885128 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet dbfe80d388644d0488cb2431a334c908 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:28.886880 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b7d85a2941545ceaa16b22e8328b4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:28.881752 1945 tablet_service.cc:1505] Processing CreateTablet for tablet b8fe7825fa8d46669772cf84bc4b3afc (DEFAULT_TABLE table=default.Table_c1abfb93d24a4f2a97d8d87a06c34ea5 [id=649f153da3b14f0b99acef230e010a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:28.890301 2372 ts_tablet_manager.cc:1929] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:28.890439 2372 log.cc:1199] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/aa399e6407fd4c4d9871d3d5ccf894ec
I20251026 02:56:28.890822 2372 ts_tablet_manager.cc:1950] T aa399e6407fd4c4d9871d3d5ccf894ec P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:28.905887 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 362) took 1239 ms. Trace:
I20251026 02:56:28.906337 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66ede86c604e47dbb982e5a3ba18bcb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.907284 2372 rpcz_store.cc:276] 1026 02:56:27.666148 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.666996 (+ 848us) service_pool.cc:225] Handling call
1026 02:56:28.762808 (+1095812us) ts_tablet_manager.cc:1151] Deleting tablet aa399e6407fd4c4d9871d3d5ccf894ec
1026 02:56:28.762820 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:28.867139 (+104319us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.890140 (+ 23001us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.905830 (+ 15690us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5504,"tablet-delete.queue_time_us":1095528}
I20251026 02:56:28.907853 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet aa399e6407fd4c4d9871d3d5ccf894ec (table default.Table_65488c1f07e542b5973be5cfd122787c [id=ce3610a9d6814ec783ae7503e58ef3ad]) successfully deleted
I20251026 02:56:28.908690 2165 heartbeater.cc:499] Master 127.1.150.60:33729 was elected leader, sending a full tablet report...
I20251026 02:56:28.908807 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 885c10070d6942f7abe52c4558d887e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.908839 2372 tablet_replica.cc:333] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:28.909119 2372 raft_consensus.cc:2243] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:28.909346 2372 raft_consensus.cc:2272] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:28.910068 2372 ts_tablet_manager.cc:1916] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:28.913434 2375 raft_consensus.cc:359] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.913793 2375 raft_consensus.cc:385] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:28.913926 2375 raft_consensus.cc:740] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:28.914144 2440 raft_consensus.cc:515] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.914292 2375 consensus_queue.cc:260] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:28.914827 2440 leader_election.cc:290] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:28.916313 2194 consensus_peers.cc:597] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:28.916989 2195 leader_election.cc:304] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
W20251026 02:56:28.921186 2194 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 54284.3us
I20251026 02:56:28.922986 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79c18ebf56e045b6a52fae8871a9107f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:28.923889 2194 leader_election.cc:343] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:28.926859 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 78417us
W20251026 02:56:28.927131 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 548a7f9628ac4623b2c2f723f83c1e45 consensus state has no leader
I20251026 02:56:28.929369 2375 ts_tablet_manager.cc:1434] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.137s user 0.001s sys 0.001s
W20251026 02:56:28.931565 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fb42b54f558499090a1d7cf257dd84c consensus state has no leader
I20251026 02:56:28.931834 2631 leader_election.cc:290] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:28.932132 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c966cae6886048a2b6ad3b8bd7007bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.932021 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a59579db048a4eec8a0a47544626d25f" } new_table_name: "default.Table_056ac0729a274fb19574d75c2f37511b" modify_external_catalogs: true
I20251026 02:56:28.933871 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_79819a4391244437bf99313c2c857657"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:28.934484 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79819a4391244437bf99313c2c857657 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:28.935428 2440 raft_consensus.cc:2804] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251026 02:56:28.935807 2195 leader_election.cc:343] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: Tablet not RUNNING: STOPPED
I20251026 02:56:28.936336 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4293bc067ecf4838a1557178f2c53ec1" } modify_external_catalogs: true
I20251026 02:56:28.936493 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4293bc067ecf4838a1557178f2c53ec1" } modify_external_catalogs: true
I20251026 02:56:28.938946 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbfe80d388644d0488cb2431a334c908" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:28.939204 2114 raft_consensus.cc:2468] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:28.939508 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22e4a433801a4cb58062d3e76c760b69" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:28.939692 2118 raft_consensus.cc:3060] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:28.943135 2118 raft_consensus.cc:2468] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:28.944150 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 155a23a93358487a9564bb5ac175b4eb (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]) successfully deleted
I20251026 02:56:28.935577 2375 tablet_bootstrap.cc:492] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:28.946147 2375 tablet_bootstrap.cc:654] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:28.946549 2899 tablet.cc:1722] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251026 02:56:28.959468 2194 leader_election.cc:304] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:28.961269 2440 raft_consensus.cc:493] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:28.960551 1700 master_runner.cc:186] Error getting master registration for 127.1.150.60:33729: OK, instance_id { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" instance_seqno: 1761447385850427 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251026 02:56:28.961743 1700 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251026 02:56:28.962067 2195 consensus_peers.cc:597] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8dbc12d49b094dbeadad2e76916ba9af. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:28.963119 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet dbfe80d388644d0488cb2431a334c908 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:28.963850 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet dfd3e23d83a84648bd11cd258377aecb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:28.965910 2381 tablet_bootstrap.cc:492] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:28.965911 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b7d85a2941545ceaa16b22e8328b4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:28.967098 2933 tablet.cc:1722] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251026 02:56:28.967726 2631 raft_consensus.cc:2749] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:28.969007 2195 consensus_peers.cc:597] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7845ebd3e9ca4897b8fb19a48a010315. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:28.969285 2195 leader_election.cc:304] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:28.969302 2921 raft_consensus.cc:493] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 696476265495492790739ec126a08539)
I20251026 02:56:28.969511 2921 raft_consensus.cc:515] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:28.970813 2194 leader_election.cc:343] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:28.976646 2372 ts_tablet_manager.cc:1929] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:28.976992 2372 log.cc:1199] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/78a624d839934d3089b64e397d4360af
I20251026 02:56:28.976401 2195 leader_election.cc:304] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:28.981711 2631 raft_consensus.cc:2804] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:28.981860 2631 raft_consensus.cc:493] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:28.983686 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 332) took 1552 ms. Trace:
I20251026 02:56:28.984300 1949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 109073eb664d4090b7a2b786146e13fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:28.984577 2381 ts_tablet_manager.cc:1403] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.329s user 0.006s sys 0.011s
I20251026 02:56:28.984882 2927 tablet.cc:1722] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:28.998890 2894 raft_consensus.cc:2804] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:28.999104 2894 raft_consensus.cc:697] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
W20251026 02:56:29.000739 2194 leader_election.cc:343] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.000914 2894 consensus_queue.cc:237] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:29.001204 2194 leader_election.cc:343] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.001350 2194 leader_election.cc:304] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:29.000814 2440 raft_consensus.cc:3060] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:29.002115 2631 raft_consensus.cc:3060] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:29.004356 2909 raft_consensus.cc:493] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 209968db4dee47c49497a66c3ee750b7)
I20251026 02:56:29.004535 2720 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34746 (request call id 0) took 1593 ms. Trace:
I20251026 02:56:29.004626 2909 raft_consensus.cc:515] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:29.005903 2849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209968db4dee47c49497a66c3ee750b7. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:29.006271 2849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:29.009915 2909 leader_election.cc:290] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 209968db4dee47c49497a66c3ee750b7 (127.1.150.62:44525), 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655)
I20251026 02:56:29.013381 2892 raft_consensus.cc:2749] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:29.020402 2260 raft_consensus.cc:2468] T 47bfff40c181432896d6898ea50cd025 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:29.030247 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 892a5fab2af1432eb31d92d9859ea2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.032440 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet dfd3e23d83a84648bd11cd258377aecb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.032856 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet dbfe80d388644d0488cb2431a334c908 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:29.039070 2195 consensus_peers.cc:597] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c570e5f9a93c45bda32ec92c68794714. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:29.039522 2053 leader_election.cc:304] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:29.040148 2934 raft_consensus.cc:2804] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:29.040381 2934 raft_consensus.cc:697] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:29.040759 2934 consensus_queue.cc:237] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.047052 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22e4a433801a4cb58062d3e76c760b69" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:29.032845 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbfe80d388644d0488cb2431a334c908" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:29.049660 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "740f7c7f186f45ceaa4c40e76419e3a1" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:29.050273 1947 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b7d85a2941545ceaa16b22e8328b4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:28.983851 2376 rpcz_store.cc:276] 1026 02:56:27.414019 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.414068 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:28.053088 (+639020us) ts_tablet_manager.cc:1151] Deleting tablet 8667ffa25f2b478a987826d486400bbe
1026 02:56:28.053096 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:28.400203 (+347107us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.859089 (+458886us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.888973 (+ 29884us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x6160001c0730. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc42739dde9 00007fc429f5f5eb 00007fc429f922d4 00007fc429f72d70 00007fc429f72afd 0000000000a8867c 00007fc41a1d5d7f 00007fc41a1d904b 00007fc41a1d8ec8 0000000000a8867c
1026 02:56:28.966464 (+ 77491us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2534272,"tablet-delete.queue_time_us":624760}
I20251026 02:56:29.053378 2921 leader_election.cc:290] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:29.054623 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 335) took 1573 ms. Trace:
I20251026 02:56:29.054751 2376 rpcz_store.cc:276] 1026 02:56:27.480773 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.489527 (+ 8754us) service_pool.cc:225] Handling call
1026 02:56:29.051575 (+1562048us) ts_tablet_manager.cc:1151] Deleting tablet ec13c5386abe414ba25589e03d5b850a
1026 02:56:29.051590 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.054115 (+ 2525us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1561684}
I20251026 02:56:29.055476 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 885c10070d6942f7abe52c4558d887e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.056411 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 338) took 1574 ms. Trace:
I20251026 02:56:29.056524 2376 rpcz_store.cc:276] 1026 02:56:27.481060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.498010 (+ 16950us) service_pool.cc:225] Handling call
1026 02:56:29.055046 (+1557036us) ts_tablet_manager.cc:1151] Deleting tablet a012d868a78f4324a92db83c81931b9d
1026 02:56:29.055059 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.055163 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1556792}
I20251026 02:56:29.057638 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 337) took 1575 ms. Trace:
I20251026 02:56:29.057739 2376 rpcz_store.cc:276] 1026 02:56:27.480966 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.532772 (+ 51806us) service_pool.cc:225] Handling call
1026 02:56:29.056752 (+1523980us) ts_tablet_manager.cc:1151] Deleting tablet a6f9ae327aa4401baf66a737eaa93cf6
1026 02:56:29.056762 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.056876 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519252}
I20251026 02:56:29.058504 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 336) took 1577 ms. Trace:
I20251026 02:56:29.058607 2376 rpcz_store.cc:276] 1026 02:56:27.480857 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.540323 (+ 59466us) service_pool.cc:225] Handling call
1026 02:56:29.057969 (+1517646us) ts_tablet_manager.cc:1151] Deleting tablet 741792cf96164695b03de3fb86b12602
1026 02:56:29.057978 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.058057 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1517412}
I20251026 02:56:29.058447 2952 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251026 02:56:29.067288 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8fe7825fa8d46669772cf84bc4b3afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.067759 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "740f7c7f186f45ceaa4c40e76419e3a1" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:29.068038 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 339) took 1520 ms. Trace:
I20251026 02:56:29.068138 2376 rpcz_store.cc:276] 1026 02:56:27.547256 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.547320 (+ 64us) service_pool.cc:225] Handling call
1026 02:56:29.058827 (+1511507us) ts_tablet_manager.cc:1151] Deleting tablet 8c1e63ac8c5645f3ba1578792c0a0dae
1026 02:56:29.058836 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.067216 (+ 8380us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1511286}
I20251026 02:56:29.068987 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c966cae6886048a2b6ad3b8bd7007bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.070063 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81775055d98349f696cdab6171ffd58b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.070631 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet be1c0948c0ae4563b7cb644521914cb9
I20251026 02:56:29.072115 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24449cb9121642009dec41220e27b193. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.072700 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79c18ebf56e045b6a52fae8871a9107f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.073035 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (table default.Table_c9c7b9787c3a458ea41be132b0841599 [id=1d09f5b9fa164ac783f0be9332eac4c3]). Expected version 5 got 4
I20251026 02:56:29.073480 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ac38e6257e74367b06b15fc49d2af81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.074211 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b5ccd4d8815f49b192cb9f1973042cce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:29.074931 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5ccd4d8815f49b192cb9f1973042cce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:29.076422 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 276bd3b97c97426daf802e1f57feb265 consensus state has no leader
I20251026 02:56:29.076756 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aea01a248b345cf81d4d9862744eb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.076829 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3107109fa3984828a1e2651e63ffb1ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:29.077327 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3107109fa3984828a1e2651e63ffb1ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.077616 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7834a85168364390a841fc883681c845. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.077767 2440 raft_consensus.cc:515] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.078150 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3af2695c41a4439bc4092b7cb1b2edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.078845 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_76d502990a87496d9d6d2d163e712939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:29.079435 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76d502990a87496d9d6d2d163e712939 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.080760 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 8667ffa25f2b478a987826d486400bbe (table default.Table_4fd5f39751624015ad57dd7fc77ce18a [id=f7f1e852a46f4129b2709fd58a6be034]) successfully deleted
I20251026 02:56:29.080937 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_00dc4c49a19246889353df2157eebf15" } new_table_name: "default.Table_016812daf097412081aff793ad75f72a" modify_external_catalogs: true
I20251026 02:56:29.080942 2631 raft_consensus.cc:515] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:29.081444 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ec13c5386abe414ba25589e03d5b850a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec13c5386abe414ba25589e03d5b850a already in progress: creating tablet
I20251026 02:56:29.081534 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9cfeb6e20450444da7688fe216f7a5c0" } modify_external_catalogs: true
I20251026 02:56:29.081669 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9cfeb6e20450444da7688fe216f7a5c0" } modify_external_catalogs: true
I20251026 02:56:29.082221 2372 ts_tablet_manager.cc:1950] T 78a624d839934d3089b64e397d4360af P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:29.082326 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet a012d868a78f4324a92db83c81931b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a012d868a78f4324a92db83c81931b9d already in progress: creating tablet
I20251026 02:56:29.065912 2381 raft_consensus.cc:359] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.086619 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4011930a277d410f863a1558416c4b0f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:29.086869 2227 tablet_service.cc:1552] Processing DeleteTablet for tablet be1c0948c0ae4563b7cb644521914cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:29.087716 2114 raft_consensus.cc:3060] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:29.088351 2194 leader_election.cc:343] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.090052 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 367) took 1339 ms. Trace:
I20251026 02:56:29.090215 2440 leader_election.cc:290] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:29.090157 2372 rpcz_store.cc:276] 1026 02:56:27.748132 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.748194 (+ 62us) service_pool.cc:225] Handling call
1026 02:56:28.908764 (+1160570us) ts_tablet_manager.cc:1151] Deleting tablet 78a624d839934d3089b64e397d4360af
1026 02:56:28.908779 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:28.952450 (+ 43671us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:28.976481 (+ 24031us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.087310 (+110829us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1159957}
I20251026 02:56:29.091502 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet be1c0948c0ae4563b7cb644521914cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.070552 1915 leader_election.cc:304] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
W20251026 02:56:29.092902 2194 leader_election.cc:343] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.094645 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 78a624d839934d3089b64e397d4360af (table default.Table_a72502f837fb4542b1f00bab5da37724 [id=4db9b61c88b5422fade5d42038bc5bcf]) successfully deleted
I20251026 02:56:29.095034 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 368) took 1333 ms. Trace:
W20251026 02:56:29.096438 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet a6f9ae327aa4401baf66a737eaa93cf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6f9ae327aa4401baf66a737eaa93cf6 already in progress: creating tablet
W20251026 02:56:29.096854 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 741792cf96164695b03de3fb86b12602 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 741792cf96164695b03de3fb86b12602 already in progress: creating tablet
I20251026 02:56:29.100317 1656 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_00dc4c49a19246889353df2157eebf15 [id=347d712aca4e477c89805846800fb824] alter complete (version 1)
I20251026 02:56:29.100317 2372 rpcz_store.cc:276] 1026 02:56:27.756839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.756941 (+ 102us) service_pool.cc:225] Handling call
1026 02:56:29.090469 (+1333528us) ts_tablet_manager.cc:1151] Deleting tablet f437d8b5f5784e8fbec10d0700e90348
1026 02:56:29.090479 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.090595 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333252}
W20251026 02:56:29.100919 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 8c1e63ac8c5645f3ba1578792c0a0dae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c1e63ac8c5645f3ba1578792c0a0dae already in progress: creating tablet
I20251026 02:56:29.100879 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_80ca368a0ade4ac2b090b31b2ab56f12" } modify_external_catalogs: true
I20251026 02:56:29.101078 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_80ca368a0ade4ac2b090b31b2ab56f12" } modify_external_catalogs: true
I20251026 02:56:29.101339 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 363) took 1421 ms. Trace:
I20251026 02:56:29.101436 2372 rpcz_store.cc:276] 1026 02:56:27.679440 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.762167 (+ 82727us) service_pool.cc:225] Handling call
1026 02:56:29.100689 (+1338522us) ts_tablet_manager.cc:1151] Deleting tablet 3c3050e72d42417ab58a05f7e3aa56b3
1026 02:56:29.100701 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.100809 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1338142}
I20251026 02:56:29.101596 1947 tablet_service.cc:1552] Processing DeleteTablet for tablet be1c0948c0ae4563b7cb644521914cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:29.101656 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4b0933f71cb4ac7a58986d16b613e64 consensus state has no leader
I20251026 02:56:29.102047 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_85c523c971b34e68bed4afa5d1aac450" } modify_external_catalogs: true
I20251026 02:56:29.102164 2631 leader_election.cc:290] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:29.102191 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_85c523c971b34e68bed4afa5d1aac450" } modify_external_catalogs: true
I20251026 02:56:29.102372 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 377) took 1221 ms. Trace:
I20251026 02:56:29.102453 2372 rpcz_store.cc:276] 1026 02:56:27.879828 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.880872 (+ 1044us) service_pool.cc:225] Handling call
1026 02:56:29.101668 (+1220796us) ts_tablet_manager.cc:1151] Deleting tablet 9c7efb70c0804c149f36440a4a9e6ef8
1026 02:56:29.101679 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.101779 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1217216}
I20251026 02:56:29.103870 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 381) took 1007 ms. Trace:
I20251026 02:56:29.103967 2372 rpcz_store.cc:276] 1026 02:56:28.095395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.096055 (+ 660us) service_pool.cc:225] Handling call
1026 02:56:29.102689 (+1006634us) ts_tablet_manager.cc:1151] Deleting tablet 86e483c52e514f1da54335479f59d68e
1026 02:56:29.102698 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.102777 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":977414}
W20251026 02:56:29.104666 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet f437d8b5f5784e8fbec10d0700e90348 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f437d8b5f5784e8fbec10d0700e90348 already in progress: creating tablet
W20251026 02:56:29.104990 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9c7efb70c0804c149f36440a4a9e6ef8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c7efb70c0804c149f36440a4a9e6ef8 already in progress: creating tablet
W20251026 02:56:29.105172 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 86e483c52e514f1da54335479f59d68e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86e483c52e514f1da54335479f59d68e already in progress: creating tablet
W20251026 02:56:29.105345 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 122de8616b8f4fcf88971d5738c47e82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 122de8616b8f4fcf88971d5738c47e82 already in progress: creating tablet
W20251026 02:56:29.105542 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet aa10cd990a944eab8d9eb66aaf20080c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa10cd990a944eab8d9eb66aaf20080c already in progress: creating tablet
W20251026 02:56:29.105710 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 17d2cc6b16d34331bb80c576c0cd2ce0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17d2cc6b16d34331bb80c576c0cd2ce0 already in progress: creating tablet
W20251026 02:56:29.105887 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a53f25688e3b45dca68984e2c543d094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a53f25688e3b45dca68984e2c543d094 already in progress: creating tablet
I20251026 02:56:29.106004 2372 tablet_replica.cc:333] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:29.106052 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 209fa71af13142a58efb9f3dab3eecd3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 209fa71af13142a58efb9f3dab3eecd3 already in progress: creating tablet
W20251026 02:56:29.106213 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet efd751ee4b1d433198dcd00c081f648d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efd751ee4b1d433198dcd00c081f648d already in progress: creating tablet
I20251026 02:56:29.106220 2372 raft_consensus.cc:2243] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:29.106405 2114 raft_consensus.cc:2468] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:29.106420 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet c570e5f9a93c45bda32ec92c68794714 because the tablet was not found. No further retry: Not found: Tablet not found: c570e5f9a93c45bda32ec92c68794714
I20251026 02:56:29.111578 2372 raft_consensus.cc:2272] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:29.112195 2372 ts_tablet_manager.cc:1916] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:29.114490 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbfe80d388644d0488cb2431a334c908" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:29.114396 2377 ts_tablet_manager.cc:1929] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:29.114671 2114 raft_consensus.cc:3060] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:29.114744 2377 log.cc:1199] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/c1b6257a3c8c4e1a8144ea45ff2a2a96
I20251026 02:56:29.115392 1977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4011930a277d410f863a1558416c4b0f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:29.115480 2195 consensus_peers.cc:597] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9156314f98ac4ed9981f6f13fc585043. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:29.116006 1977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbfe80d388644d0488cb2431a334c908" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:29.122406 2114 raft_consensus.cc:2468] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:29.125896 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9ec035b1caac450d9ab55e74a061ae38
I20251026 02:56:29.127264 2195 leader_election.cc:304] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:29.128409 2631 raft_consensus.cc:2804] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:29.129356 2375 tablet_bootstrap.cc:492] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:29.129503 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ec035b1caac450d9ab55e74a061ae38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.129582 2375 ts_tablet_manager.cc:1403] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.194s user 0.005s sys 0.010s
W20251026 02:56:29.130218 2194 leader_election.cc:343] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.130467 2375 raft_consensus.cc:359] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.130637 2375 raft_consensus.cc:385] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:29.130757 2375 raft_consensus.cc:740] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:29.131078 2377 ts_tablet_manager.cc:1950] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:29.131033 2375 consensus_queue.cc:260] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.131431 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 314) took 2396 ms. Trace:
I20251026 02:56:29.132037 2377 rpcz_store.cc:276] 1026 02:56:26.734849 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.734910 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:28.829961 (+2095051us) ts_tablet_manager.cc:1151] Deleting tablet c1b6257a3c8c4e1a8144ea45ff2a2a96
1026 02:56:28.829986 (+ 25us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.021994 (+192008us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.107263 (+ 85269us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.131399 (+ 24136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2094832}
I20251026 02:56:29.132812 2377 tablet_replica.cc:333] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:29.132900 2378 ts_tablet_manager.cc:1434] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.591s user 0.015s sys 0.024s
I20251026 02:56:29.133030 2377 raft_consensus.cc:2243] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:29.133023 2378 ts_tablet_manager.cc:1467] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8: Tablet startup took 590ms
I20251026 02:56:29.133198 2377 raft_consensus.cc:2272] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:29.133154 2378 ts_tablet_manager.cc:1469] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8: Trace:
1026 02:56:22.941719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:22.941979 (+ 260us) service_pool.cc:225] Handling call
1026 02:56:22.942470 (+ 491us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:22.942496 (+ 26us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:22.959491 (+ 16995us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:23.025158 (+ 65667us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:23.025329 (+ 171us) inbound_call.cc:173] Queueing success response
1026 02:56:28.191871 (+5166542us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1026 02:56:28.542344 (+350473us) ts_tablet_manager.cc:1435] Starting tablet replica
1026 02:56:28.773619 (+231275us) tablet_replica.cc:261] Starting instrumentation
1026 02:56:28.794977 (+ 21358us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":20897,"raft.run_cpu_time_us":47,"raft.run_wall_time_us":47,"tablet-open.queue_time_us":5166584,"thread_start_us":279735,"threads_started":1}
I20251026 02:56:29.133764 2377 ts_tablet_manager.cc:1916] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:29.141520 2631 raft_consensus.cc:697] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:29.141882 2631 consensus_queue.cc:237] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.141498 2195 leader_election.cc:304] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:29.143160 2440 raft_consensus.cc:2804] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:29.143345 2440 raft_consensus.cc:697] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:29.143465 2381 raft_consensus.cc:385] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:29.143543 2381 raft_consensus.cc:740] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:29.143599 2440 consensus_queue.cc:237] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.143764 2381 consensus_queue.cc:260] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.144402 2381 ts_tablet_manager.cc:1434] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.160s user 0.000s sys 0.002s
I20251026 02:56:29.144740 2381 tablet_bootstrap.cc:492] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:29.145347 2194 leader_election.cc:343] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.145849 2923 raft_consensus.cc:493] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:29.146037 2923 raft_consensus.cc:515] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.146752 2923 leader_election.cc:290] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:29.148829 2944 raft_consensus.cc:2749] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:29.152185 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 (table default.Table_d23ac2c9657b4adeaac9545d5bc9ae41 [id=9e897d33e6374ad4bc3c8acaae512645]) successfully deleted
I20251026 02:56:29.155567 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca4611c0a494425ca50d599b310872c8" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:29.156142 2226 tablet_service.cc:1505] Processing CreateTablet for tablet f8f0124f125c40d291c2211fd95c71c5 (DEFAULT_TABLE table=default.Table_8f3c2b96b146407f87972ce3efb7e27d [id=34f4aaca15a44a109a8623dc3ea9e7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.156570 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f0124f125c40d291c2211fd95c71c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.161525 2232 tablet_service.cc:1505] Processing CreateTablet for tablet b1b6613742a24c21a1db6ebe858a846f (DEFAULT_TABLE table=default.Table_09b68d3ebab94c55b3c90567af20db3d [id=d35f10fbbafe4c1595eaf1a17c4c091e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.162051 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1b6613742a24c21a1db6ebe858a846f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:29.163453 2195 leader_election.cc:343] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.163621 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db94738da3554fbda64c23fde3ef6c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.164520 2933 tablet.cc:1722] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:29.168087 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31b6e08597bd450eb1875201cf594a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.175721 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.180367 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 868dbbfce40c4dd6b36cc220966b1986. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.182404 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 340) took 1620 ms. Trace:
I20251026 02:56:29.182526 2376 rpcz_store.cc:276] 1026 02:56:27.561673 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.561755 (+ 82us) service_pool.cc:225] Handling call
1026 02:56:29.068929 (+1507174us) ts_tablet_manager.cc:1151] Deleting tablet cfc3aed64ad0439cb22fd6815c973d87
1026 02:56:29.182201 (+113272us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.182338 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1505563}
I20251026 02:56:29.186396 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 342) took 1599 ms. Trace:
I20251026 02:56:29.186517 2376 rpcz_store.cc:276] 1026 02:56:27.586861 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.586920 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:29.182778 (+1595858us) ts_tablet_manager.cc:1151] Deleting tablet d43b6f22dd4f4f9c9657f21514550312
1026 02:56:29.186212 (+ 3434us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.186338 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1571493}
I20251026 02:56:29.187033 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 344) took 1510 ms. Trace:
I20251026 02:56:29.187126 2376 rpcz_store.cc:276] 1026 02:56:27.676819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.676879 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:29.186897 (+1510018us) ts_tablet_manager.cc:1151] Deleting tablet 3c3050e72d42417ab58a05f7e3aa56b3
1026 02:56:29.186904 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.186979 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1509477}
I20251026 02:56:29.187407 2376 tablet_replica.cc:333] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:29.187588 2376 raft_consensus.cc:2243] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:29.187772 2376 raft_consensus.cc:2272] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:29.190311 2376 ts_tablet_manager.cc:1916] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:29.191941 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f16e40a30c41788a6f9813c235c5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.193884 2378 tablet_bootstrap.cc:492] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:29.197525 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ec035b1caac450d9ab55e74a061ae38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:29.205284 2381 tablet_bootstrap.cc:654] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:29.111246 2695 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" instance_seqno: 1761447379602867) as {username='slave'} at 127.1.150.2:49587; Asking this server to re-register.
I20251026 02:56:29.189908 2705 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" instance_seqno: 1761447379108470) as {username='slave'} at 127.1.150.1:36231; Asking this server to re-register.
I20251026 02:56:29.206758 2899 tablet.cc:1722] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:29.209364 2165 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:29.213281 2705 ts_manager.cc:194] Registered new tserver with Master: 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:29.152716 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f07a93acca3840ae8da80a0cc93ca97e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:29.217979 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 65801us
W20251026 02:56:29.218461 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f07a93acca3840ae8da80a0cc93ca97e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:29.218775 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet cfc3aed64ad0439cb22fd6815c973d87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfc3aed64ad0439cb22fd6815c973d87 already in progress: creating tablet
W20251026 02:56:29.219058 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet d43b6f22dd4f4f9c9657f21514550312 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43b6f22dd4f4f9c9657f21514550312 already in progress: creating tablet
I20251026 02:56:29.153110 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_20e0d029dadf4384b1efbfbc372f4741"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:29.153419 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ad7f0263e7ad4bdea1b9805522ec23db" } new_table_name: "default.Table_b6138cb4ba3c4b70bcec42f1ce989668" modify_external_catalogs: true
I20251026 02:56:29.221153 2375 ts_tablet_manager.cc:1434] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.091s user 0.001s sys 0.001s
I20251026 02:56:29.221546 2375 tablet_bootstrap.cc:492] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:29.221635 2098 tablet_service.cc:1505] Processing CreateTablet for tablet f8f0124f125c40d291c2211fd95c71c5 (DEFAULT_TABLE table=default.Table_8f3c2b96b146407f87972ce3efb7e27d [id=34f4aaca15a44a109a8623dc3ea9e7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.222071 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f0124f125c40d291c2211fd95c71c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.222083 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet dbf2a1ce78154415b23ab6d0efbc1f52 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:29.162604 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_10b9ad21b01e4323a1bdb23caf300803" } new_table_name: "default.Table_f8baefeba2b54cb39affb2d4f1b0bef8" modify_external_catalogs: true
I20251026 02:56:29.224529 2084 tablet_service.cc:1505] Processing CreateTablet for tablet b1b6613742a24c21a1db6ebe858a846f (DEFAULT_TABLE table=default.Table_09b68d3ebab94c55b3c90567af20db3d [id=d35f10fbbafe4c1595eaf1a17c4c091e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.224915 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1b6613742a24c21a1db6ebe858a846f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.173020 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94
I20251026 02:56:29.228804 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet 9a08e8b7eb1949408f2151f2d7257e9d from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:29.231221 2897 raft_consensus.cc:493] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:29.231370 2897 raft_consensus.cc:515] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.232038 2897 leader_election.cc:290] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:29.232733 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.173272 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b1ca5b8379c94deab1130bfe8bdc87b0
I20251026 02:56:29.173486 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a08efd37968846d2958e54831488235c
W20251026 02:56:29.234946 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20e0d029dadf4384b1efbfbc372f4741 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.173583 1654 catalog_manager.cc:5649] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:29.243599 2377 ts_tablet_manager.cc:1929] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:29.243903 2377 log.cc:1199] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/ac19c5a847454a23ab16831198ff2a20
I20251026 02:56:29.244789 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:29.250480 2377 ts_tablet_manager.cc:1950] T ac19c5a847454a23ab16831198ff2a20 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:29.253196 2899 tablet.cc:1722] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:29.254859 2934 raft_consensus.cc:493] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:29.255113 2934 raft_consensus.cc:515] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.256465 1673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 4 candidate_status { last_received { term: 3 index: 407 } } ignore_live_leader: false dest_uuid: "209968db4dee47c49497a66c3ee750b7" is_pre_election: true
I20251026 02:56:29.257030 1661 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 740f7c7f186f45ceaa4c40e76419e3a1 (table default.Table_c9c7b9787c3a458ea41be132b0841599 [id=1d09f5b9fa164ac783f0be9332eac4c3]). Expected version 5 got 0
W20251026 02:56:29.257210 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 548a7f9628ac4623b2c2f723f83c1e45 consensus state has no leader
W20251026 02:56:29.257851 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c31d2704d1f14ddead69c794cc631ec1 consensus state has no leader
I20251026 02:56:29.258085 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet a08efd37968846d2958e54831488235c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.258626 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet dbf2a1ce78154415b23ab6d0efbc1f52
I20251026 02:56:29.259131 1657 catalog_manager.cc:6719] ReplaceTablet: tablet dbf2a1ce78154415b23ab6d0efbc1f52 deleted and replaced by tablet f51f8fef342b46abafff5bf12dba7184
I20251026 02:56:29.260231 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9a08e8b7eb1949408f2151f2d7257e9d
I20251026 02:56:29.260748 1660 catalog_manager.cc:6719] ReplaceTablet: tablet 9a08e8b7eb1949408f2151f2d7257e9d deleted and replaced by tablet 1a1b8c440061429caea0f0803f110742
I20251026 02:56:29.261409 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet dbf2a1ce78154415b23ab6d0efbc1f52 with delete_type TABLET_DATA_DELETED (replaced by tablet f51f8fef342b46abafff5bf12dba7184) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:29.261703 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet ce922040298d4952af13652eeda7a234 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:29.263365 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a08e8b7eb1949408f2151f2d7257e9d with delete_type TABLET_DATA_DELETED (replaced by tablet 1a1b8c440061429caea0f0803f110742) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:29.264278 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet b1ca5b8379c94deab1130bfe8bdc87b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.265223 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet dbf2a1ce78154415b23ab6d0efbc1f52 with delete_type TABLET_DATA_DELETED (replaced by tablet f51f8fef342b46abafff5bf12dba7184) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.266522 2375 tablet_bootstrap.cc:654] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:29.264869 2934 leader_election.cc:290] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:29.267696 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a08e8b7eb1949408f2151f2d7257e9d with delete_type TABLET_DATA_DELETED (replaced by tablet 1a1b8c440061429caea0f0803f110742) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.245656 2381 tablet_bootstrap.cc:492] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:29.245821 2225 tablet_service.cc:1552] Processing DeleteTablet for tablet b1ca5b8379c94deab1130bfe8bdc87b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:29.249114 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet a08efd37968846d2958e54831488235c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:29.269616 2381 ts_tablet_manager.cc:1403] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.125s user 0.000s sys 0.004s
I20251026 02:56:29.270859 2381 raft_consensus.cc:359] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.277349 2381 raft_consensus.cc:385] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:29.277690 2381 raft_consensus.cc:740] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:29.278505 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a7b1603151c54757bd2505c7958e3bae" } modify_external_catalogs: true
I20251026 02:56:29.278748 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a7b1603151c54757bd2505c7958e3bae" } modify_external_catalogs: true
I20251026 02:56:29.278275 2381 consensus_queue.cc:260] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.280715 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_73433427e5f34844932d213dfb63f8f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:29.281451 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73433427e5f34844932d213dfb63f8f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.282079 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_47d5f6b5a01b4d4d991fca2b89af298e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:29.282982 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cba8c65ea6a14b09a43a61d89e0ecee4" } new_table_name: "default.Table_71583d73a49142ae9fec31193fb53542" modify_external_catalogs: true
W20251026 02:56:29.283098 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47d5f6b5a01b4d4d991fca2b89af298e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.283597 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_500c2af7af964a57bc2b5c219aadaeb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:29.284157 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_500c2af7af964a57bc2b5c219aadaeb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.285733 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c9ed6201de674968b4f3b40267527312"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:29.286358 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9ed6201de674968b4f3b40267527312 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.291107 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet bfa68f2c942c43e2a02e9d333bdfa2a2 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:29.296149 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 82634us
I20251026 02:56:29.303881 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_840a6bd369ab416191c1f0ee15bd50ef" } new_table_name: "default.Table_0fb1c918f03c49af90f8630333b5bab7" modify_external_catalogs: true
I20251026 02:56:29.305241 2381 ts_tablet_manager.cc:1434] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20251026 02:56:29.309265 2897 raft_consensus.cc:493] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:29.310036 2195 consensus_peers.cc:597] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8533f1c191042ee9f2c0b99972fe931. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:29.309993 2897 raft_consensus.cc:515] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.313995 2897 leader_election.cc:290] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:29.323165 2372 ts_tablet_manager.cc:1929] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:29.323460 2372 log.cc:1199] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/a70d70abc3a34dac8cf9a747903fa78c
I20251026 02:56:29.323995 2372 ts_tablet_manager.cc:1950] T a70d70abc3a34dac8cf9a747903fa78c P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:29.324295 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ce922040298d4952af13652eeda7a234
I20251026 02:56:29.325232 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet a70d70abc3a34dac8cf9a747903fa78c (table default.Table_c4bf189953554cb8854613da705a2b42 [id=a84628b0b09f43c69e86344d40641b02]) successfully deleted
I20251026 02:56:29.325553 1658 catalog_manager.cc:6719] ReplaceTablet: tablet ce922040298d4952af13652eeda7a234 deleted and replaced by tablet e830f1aa1270420c9b28a0491a1a0960
I20251026 02:56:29.325464 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet ce922040298d4952af13652eeda7a234 with delete_type TABLET_DATA_DELETED (replaced by tablet e830f1aa1270420c9b28a0491a1a0960) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.325928 2372 tablet_replica.cc:333] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:29.326368 2372 raft_consensus.cc:2243] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:29.327371 2372 raft_consensus.cc:2272] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:29.330855 2372 ts_tablet_manager.cc:1916] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:29.332947 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet ce922040298d4952af13652eeda7a234 with delete_type TABLET_DATA_DELETED (replaced by tablet e830f1aa1270420c9b28a0491a1a0960) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:29.342149 2375 tablet_bootstrap.cc:492] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:29.342374 2375 ts_tablet_manager.cc:1403] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.121s user 0.005s sys 0.001s
I20251026 02:56:29.343322 2375 raft_consensus.cc:359] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.343621 2375 raft_consensus.cc:385] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:29.343722 2375 raft_consensus.cc:740] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:29.344030 2375 consensus_queue.cc:260] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.344573 2375 ts_tablet_manager.cc:1434] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251026 02:56:29.345185 1654 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_c9c7b9787c3a458ea41be132b0841599 [id=1d09f5b9fa164ac783f0be9332eac4c3] alter complete (version 5)
I20251026 02:56:29.345264 2375 tablet_bootstrap.cc:492] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:29.357453 2375 tablet_bootstrap.cc:654] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:29.362032 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b120712c89884acc8703a30980f5a265" } modify_external_catalogs: true
I20251026 02:56:29.362264 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b120712c89884acc8703a30980f5a265" } modify_external_catalogs: true
I20251026 02:56:29.360718 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet b1a2522801364693be0bda6fa45a697e with delete_type TABLET_DATA_DELETED (replaced by tablet ff242ef9541a42d793c4484190956da6) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.363914 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a72c8f4a62014bbeb2bc7cbb63991756"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:29.364836 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a72c8f4a62014bbeb2bc7cbb63991756 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.372419 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet f2d97084b5914350afbfda5c2a34bf9a
I20251026 02:56:29.374153 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet f2d97084b5914350afbfda5c2a34bf9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:29.374745 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet f2d97084b5914350afbfda5c2a34bf9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.374791 2372 ts_tablet_manager.cc:1929] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:29.387889 2372 log.cc:1199] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/e62064fbdffb4954a3a1227efd6327ee
I20251026 02:56:29.388532 2372 ts_tablet_manager.cc:1950] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:29.389629 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet e62064fbdffb4954a3a1227efd6327ee (table default.Table_e9da6904da2c4d15bb17df7e9bbb3221 [id=fb1dbbc49ad94d85925be7fc850041e0]) successfully deleted
I20251026 02:56:29.391155 2372 tablet_replica.cc:333] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:29.391532 2372 raft_consensus.cc:2243] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:29.391871 2372 raft_consensus.cc:2272] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:29.392581 2372 ts_tablet_manager.cc:1916] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:29.392589 2261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1a2522801364693be0bda6fa45a697e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:29.395344 2933 tablet.cc:1722] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251026 02:56:29.395445 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 41b7b0c36ba748f0a1565892dbe98a78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41b7b0c36ba748f0a1565892dbe98a78 already in progress: creating tablet
W20251026 02:56:29.395780 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet dfd3e23d83a84648bd11cd258377aecb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfd3e23d83a84648bd11cd258377aecb already in progress: creating tablet
I20251026 02:56:29.395735 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86d8e1e21c5f440d8baf76c357d01ad2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:29.399019 2705 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "696476265495492790739ec126a08539" instance_seqno: 1761447380121209) as {username='slave'} at 127.1.150.3:49387; Asking this server to re-register.
I20251026 02:56:29.406347 2308 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:29.408860 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 313) took 2674 ms. Trace:
I20251026 02:56:29.408991 2377 rpcz_store.cc:276] 1026 02:56:26.734683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.736433 (+ 1750us) service_pool.cc:225] Handling call
1026 02:56:29.132396 (+2395963us) ts_tablet_manager.cc:1151] Deleting tablet ac19c5a847454a23ab16831198ff2a20
1026 02:56:29.132407 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.138325 (+ 5918us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.243407 (+105082us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.408787 (+165380us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2395764}
I20251026 02:56:29.409472 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 315) took 2670 ms. Trace:
I20251026 02:56:29.409566 2377 rpcz_store.cc:276] 1026 02:56:26.738994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.739053 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:29.409300 (+2670247us) ts_tablet_manager.cc:1151] Deleting tablet 882fe81da21344ed99ea735fa79f73c0
1026 02:56:29.409310 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.409415 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2670057}
I20251026 02:56:29.409943 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 316) took 2575 ms. Trace:
I20251026 02:56:29.410050 2377 rpcz_store.cc:276] 1026 02:56:26.834931 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.834992 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:29.409784 (+2574792us) ts_tablet_manager.cc:1151] Deleting tablet d65b0d79e2e54af9810022b683c3288b
1026 02:56:29.409792 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.409889 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2574518}
I20251026 02:56:29.410431 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 317) took 2575 ms. Trace:
I20251026 02:56:29.410516 2377 rpcz_store.cc:276] 1026 02:56:26.835118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.835298 (+ 180us) service_pool.cc:225] Handling call
1026 02:56:29.410275 (+2574977us) ts_tablet_manager.cc:1151] Deleting tablet e62064fbdffb4954a3a1227efd6327ee
1026 02:56:29.410286 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.410377 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2574809}
I20251026 02:56:29.412472 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet ac19c5a847454a23ab16831198ff2a20 (table default.Table_f8a501269b994d8c8864349d12974e16 [id=a48fa20d35e44980b60317a9c93b42d5]) successfully deleted
W20251026 02:56:29.412750 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 882fe81da21344ed99ea735fa79f73c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 882fe81da21344ed99ea735fa79f73c0 already in progress: creating tablet
W20251026 02:56:29.412976 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet d65b0d79e2e54af9810022b683c3288b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d65b0d79e2e54af9810022b683c3288b already in progress: creating tablet
W20251026 02:56:29.413173 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet e62064fbdffb4954a3a1227efd6327ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e62064fbdffb4954a3a1227efd6327ee already in progress: creating tablet
I20251026 02:56:29.413193 2705 ts_manager.cc:194] Registered new tserver with Master: 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:29.410837 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 318) took 2570 ms. Trace:
I20251026 02:56:29.413578 2377 rpcz_store.cc:276] 1026 02:56:26.839910 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:26.839984 (+ 74us) service_pool.cc:225] Handling call
1026 02:56:29.410703 (+2570719us) ts_tablet_manager.cc:1151] Deleting tablet f85c2745a371455a8dda67f31a19c9d5
1026 02:56:29.410710 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.410780 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2570435}
I20251026 02:56:29.413877 2377 tablet_replica.cc:333] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:29.414072 2377 raft_consensus.cc:2243] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:29.414322 2377 raft_consensus.cc:2272] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:29.414877 2377 ts_tablet_manager.cc:1916] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:29.414942 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e62064fbdffb4954a3a1227efd6327ee" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:29.411340 2381 tablet_bootstrap.cc:492] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:29.417728 2381 tablet_bootstrap.cc:654] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:29.426352 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_60d87251cafc4e6ca68c5e0affce8260" } new_table_name: "default.Table_6e2b159cc3414b619a96fa2c002c764d" modify_external_catalogs: true
W20251026 02:56:29.438330 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet f85c2745a371455a8dda67f31a19c9d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f85c2745a371455a8dda67f31a19c9d5 already in progress: creating tablet
I20251026 02:56:29.442852 2927 tablet.cc:1722] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20251026 02:56:29.393203 2053 leader_election.cc:343] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:29.448473 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 55255.6us
W20251026 02:56:29.448951 2053 leader_election.cc:343] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:29.449208 2053 leader_election.cc:343] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.415449 2964 raft_consensus.cc:493] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:29.449414 2964 raft_consensus.cc:515] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.450275 2964 leader_election.cc:290] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:29.451041 2027 heartbeater.cc:461] Registering TS with master...
W20251026 02:56:29.454404 1648 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 548a7f9628ac4623b2c2f723f83c1e45 (table default.Table_f8baefeba2b54cb39affb2d4f1b0bef8 [id=d9a6b0b6f3e246a3aa8fbd1167205264], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 548a7f9628ac4623b2c2f723f83c1e45 consensus state has no leader
I20251026 02:56:29.454721 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5226d4516ce24ecda51dab35c86da132" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:29.454981 2262 raft_consensus.cc:2468] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:29.455462 2053 leader_election.cc:304] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:29.455751 2934 raft_consensus.cc:2804] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:29.455823 2934 raft_consensus.cc:493] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:29.455912 2934 raft_consensus.cc:3060] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:29.458938 2934 raft_consensus.cc:515] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.461333 2934 leader_election.cc:290] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:29.462303 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5226d4516ce24ecda51dab35c86da132" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:29.462525 2259 raft_consensus.cc:3060] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:29.465615 2259 raft_consensus.cc:2468] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:29.466265 1949 tablet_service.cc:1505] Processing CreateTablet for tablet f8f0124f125c40d291c2211fd95c71c5 (DEFAULT_TABLE table=default.Table_8f3c2b96b146407f87972ce3efb7e27d [id=34f4aaca15a44a109a8623dc3ea9e7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.466383 2053 leader_election.cc:304] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:29.466802 2934 raft_consensus.cc:2804] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:29.467576 2934 raft_consensus.cc:697] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:29.467993 2934 consensus_queue.cc:237] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.469168 1949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f0124f125c40d291c2211fd95c71c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.469498 2378 tablet_bootstrap.cc:654] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:29.472728 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ec035b1caac450d9ab55e74a061ae38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.479305 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_082149a0d40742629c17967fd62a789e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:29.480105 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_082149a0d40742629c17967fd62a789e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.482059 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet b1ca5b8379c94deab1130bfe8bdc87b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.497646 2375 tablet_bootstrap.cc:492] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:29.498438 2375 ts_tablet_manager.cc:1403] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.153s user 0.010s sys 0.031s
I20251026 02:56:29.499424 2375 raft_consensus.cc:359] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.504863 2375 raft_consensus.cc:385] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:29.505179 2375 raft_consensus.cc:740] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:29.505576 2375 consensus_queue.cc:260] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.506588 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a79373c41ad7448e903a6d939c57b345"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:29.507062 2375 ts_tablet_manager.cc:1434] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20251026 02:56:29.507241 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a79373c41ad7448e903a6d939c57b345 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.505434 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet a08efd37968846d2958e54831488235c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.507583 2375 tablet_bootstrap.cc:492] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:29.508927 2955 sasl_common.cc:323] Time spent creating new SASL kudu client: real 0.297s user 0.000s sys 0.000s
I20251026 02:56:29.509258 2375 tablet_bootstrap.cc:654] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:29.509946 2381 tablet_bootstrap.cc:492] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:29.510121 2381 ts_tablet_manager.cc:1403] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.099s user 0.005s sys 0.003s
I20251026 02:56:29.511014 2381 raft_consensus.cc:359] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.511445 2381 raft_consensus.cc:385] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:29.511521 2381 raft_consensus.cc:740] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:29.512444 2381 consensus_queue.cc:260] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.513023 2381 ts_tablet_manager.cc:1434] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 02:56:29.513383 2381 tablet_bootstrap.cc:492] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:29.515424 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 358e805370ef410e9d65ee92a8d04f34 consensus state has no leader
W20251026 02:56:29.518357 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a3f3fc3c7a04d05bae28472251f3acc consensus state has no leader
I20251026 02:56:29.518671 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet e89a98fcfdbb4011876b0ebc2af91f67 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:29.519309 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_69e73d553b1b47f6b4494421985d7889"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:29.519595 2377 ts_tablet_manager.cc:1929] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:29.519944 2377 log.cc:1199] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/ca4611c0a494425ca50d599b310872c8
W20251026 02:56:29.519942 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69e73d553b1b47f6b4494421985d7889 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.520802 2377 ts_tablet_manager.cc:1950] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:29.526175 2381 tablet_bootstrap.cc:654] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:29.533215 1952 tablet_service.cc:1552] Processing DeleteTablet for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.533332 2375 tablet_bootstrap.cc:492] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:29.533560 2375 ts_tablet_manager.cc:1403] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.001s
I20251026 02:56:29.533875 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dec83bfb9e946db90c10d526af2dfca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.534134 1744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 4 candidate_status { last_received { term: 3 index: 407 } } ignore_live_leader: false dest_uuid: "647c0de4ce394caca75179fe11658fc8" is_pre_election: true
I20251026 02:56:29.534375 2375 raft_consensus.cc:359] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.534571 2375 raft_consensus.cc:385] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:29.534682 2375 raft_consensus.cc:740] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:29.535102 2644 leader_election.cc:304] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 751b719b75ce4bc2a88a19091230ecb7; no voters: 209968db4dee47c49497a66c3ee750b7, 647c0de4ce394caca75179fe11658fc8
I20251026 02:56:29.537998 2381 tablet_bootstrap.cc:492] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:29.538151 2381 ts_tablet_manager.cc:1403] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.001s
I20251026 02:56:29.538990 2381 raft_consensus.cc:359] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.539683 2381 raft_consensus.cc:385] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:29.540257 2381 raft_consensus.cc:740] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:29.540401 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet f2d97084b5914350afbfda5c2a34bf9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.540560 2381 consensus_queue.cc:260] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.540656 1977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca4611c0a494425ca50d599b310872c8" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:29.541302 2381 ts_tablet_manager.cc:1434] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251026 02:56:29.543833 2381 tablet_bootstrap.cc:492] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:29.545676 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0cd100667df948d29f7f04a6f55b44e5" } modify_external_catalogs: true
I20251026 02:56:29.545905 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0cd100667df948d29f7f04a6f55b44e5" } modify_external_catalogs: true
I20251026 02:56:29.546073 2381 tablet_bootstrap.cc:654] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:29.546967 2375 consensus_queue.cc:260] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.547626 2375 ts_tablet_manager.cc:1434] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251026 02:56:29.547963 2375 tablet_bootstrap.cc:492] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:29.550315 2375 tablet_bootstrap.cc:654] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:29.551697 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet ce922040298d4952af13652eeda7a234 with delete_type TABLET_DATA_DELETED (replaced by tablet e830f1aa1270420c9b28a0491a1a0960) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.552580 1952 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a08e8b7eb1949408f2151f2d7257e9d with delete_type TABLET_DATA_DELETED (replaced by tablet 1a1b8c440061429caea0f0803f110742) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.553319 1946 tablet_service.cc:1505] Processing CreateTablet for tablet b1b6613742a24c21a1db6ebe858a846f (DEFAULT_TABLE table=default.Table_09b68d3ebab94c55b3c90567af20db3d [id=d35f10fbbafe4c1595eaf1a17c4c091e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:29.557619 2194 consensus_peers.cc:597] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:29.563210 2375 tablet_bootstrap.cc:492] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:29.563416 2375 ts_tablet_manager.cc:1403] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20251026 02:56:29.564292 2705 ts_manager.cc:194] Registered new tserver with Master: 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:29.564495 2375 raft_consensus.cc:359] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.564738 2375 raft_consensus.cc:385] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:29.564851 2375 raft_consensus.cc:740] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:29.565181 2375 consensus_queue.cc:260] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.566025 2375 ts_tablet_manager.cc:1434] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:29.566497 2375 tablet_bootstrap.cc:492] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:29.568009 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet dbf2a1ce78154415b23ab6d0efbc1f52 with delete_type TABLET_DATA_DELETED (replaced by tablet f51f8fef342b46abafff5bf12dba7184) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.577108 2375 tablet_bootstrap.cc:654] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:29.586735 2375 tablet_bootstrap.cc:492] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:29.586745 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 330) took 2433 ms. Trace:
I20251026 02:56:29.586936 2375 ts_tablet_manager.cc:1403] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20251026 02:56:29.586903 2377 rpcz_store.cc:276] 1026 02:56:27.153417 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.153466 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:29.413787 (+2260321us) ts_tablet_manager.cc:1151] Deleting tablet ca4611c0a494425ca50d599b310872c8
1026 02:56:29.413796 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.447765 (+ 33969us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.519430 (+ 71665us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.586695 (+ 67265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2259344}
I20251026 02:56:29.587507 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 329) took 2453 ms. Trace:
I20251026 02:56:29.587638 2377 rpcz_store.cc:276] 1026 02:56:27.133876 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.134038 (+ 162us) service_pool.cc:225] Handling call
1026 02:56:29.587293 (+2453255us) ts_tablet_manager.cc:1151] Deleting tablet ae95654d129441e08fcb3f3b355ffc22
1026 02:56:29.587307 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.587464 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2404197}
I20251026 02:56:29.587770 2375 raft_consensus.cc:359] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.587982 2375 raft_consensus.cc:385] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:29.588001 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 331) took 2398 ms. Trace:
I20251026 02:56:29.588048 2375 raft_consensus.cc:740] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:29.588104 2377 rpcz_store.cc:276] 1026 02:56:27.189221 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.189347 (+ 126us) service_pool.cc:225] Handling call
1026 02:56:29.587867 (+2398520us) ts_tablet_manager.cc:1151] Deleting tablet 5caeb11322fe48348ebb523967152187
1026 02:56:29.587875 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.587956 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2396358}
I20251026 02:56:29.588047 1661 catalog_manager.cc:5649] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:29.588326 2375 consensus_queue.cc:260] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.588435 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 333) took 2382 ms. Trace:
I20251026 02:56:29.588487 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 22e4a433801a4cb58062d3e76c760b69 (table default.Table_f7c216f3ebd34c718ceef042a435b4cd [id=b4c0611288a949378eed16d32b5170b9]). Expected version 2 got 0
I20251026 02:56:29.588539 2377 rpcz_store.cc:276] 1026 02:56:27.205994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.206334 (+ 340us) service_pool.cc:225] Handling call
1026 02:56:29.588317 (+2381983us) ts_tablet_manager.cc:1151] Deleting tablet 47037db3602246a2ab6224557286162f
1026 02:56:29.588325 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.588404 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367501}
I20251026 02:56:29.588840 2375 ts_tablet_manager.cc:1434] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:29.588896 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 332) took 2398 ms. Trace:
I20251026 02:56:29.588825 1661 catalog_manager.cc:5649] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:29.589000 2377 rpcz_store.cc:276] 1026 02:56:27.190356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.209227 (+ 18871us) service_pool.cc:225] Handling call
1026 02:56:29.588744 (+2379517us) ts_tablet_manager.cc:1151] Deleting tablet 6a3491b303f94fe2a2c838297f73d464
1026 02:56:29.588753 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.588865 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2365139}
I20251026 02:56:29.589202 2375 tablet_bootstrap.cc:492] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:29.589270 2377 tablet_replica.cc:333] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:29.589476 2377 raft_consensus.cc:2243] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:29.589781 2377 pending_rounds.cc:70] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:29.589862 2377 pending_rounds.cc:77] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888499884171264 op_type: NO_OP noop_request { }
I20251026 02:56:29.590004 2377 raft_consensus.cc:2889] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:29.590159 2377 raft_consensus.cc:2272] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:29.591315 2377 ts_tablet_manager.cc:1916] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:29.592715 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet ca4611c0a494425ca50d599b310872c8 (table default.Table_21d71fb70373400396a75b6e52416078 [id=2dcd0d98a1354635a3673182e6fad58a]) successfully deleted
W20251026 02:56:29.593010 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ae95654d129441e08fcb3f3b355ffc22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae95654d129441e08fcb3f3b355ffc22 already in progress: creating tablet
W20251026 02:56:29.593312 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 47037db3602246a2ab6224557286162f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47037db3602246a2ab6224557286162f already in progress: creating tablet
W20251026 02:56:29.593576 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 6a3491b303f94fe2a2c838297f73d464 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a3491b303f94fe2a2c838297f73d464 already in progress: creating tablet
I20251026 02:56:29.594834 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 (DEFAULT_TABLE table=default.Table_b5ccd4d8815f49b192cb9f1973042cce [id=d1ff1ff682064a5e96b158e9c64bdeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.595279 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce4092b7ccc4888b5cbe3b38bee2e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.596319 2375 tablet_bootstrap.cc:654] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:29.596724 2934 raft_consensus.cc:493] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:29.596969 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 (DEFAULT_TABLE table=default.Table_b5ccd4d8815f49b192cb9f1973042cce [id=d1ff1ff682064a5e96b158e9c64bdeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.596936 2934 raft_consensus.cc:515] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.597555 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 848e414f6e134761818039f7e98500ac (DEFAULT_TABLE table=default.Table_3107109fa3984828a1e2651e63ffb1ed [id=db65e2c421024deca4f6edc8ace95365]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.598001 2934 leader_election.cc:290] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:29.598090 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 09f5265d85c6496d85c555b2b847373a (DEFAULT_TABLE table=default.Table_76d502990a87496d9d6d2d163e712939 [id=fe6fe74d50ea4174bc20c04bc77f8215]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.598942 2226 tablet_service.cc:1505] Processing CreateTablet for tablet 848e414f6e134761818039f7e98500ac (DEFAULT_TABLE table=default.Table_3107109fa3984828a1e2651e63ffb1ed [id=db65e2c421024deca4f6edc8ace95365]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.599561 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 848e414f6e134761818039f7e98500ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.600672 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 351cd60b076a49949f03640c981845c4 (DEFAULT_TABLE table=default.Table_7ba0c9a1ed6e49f09387b3e13faee903 [id=f94002f27c954a41b8e666b012123a98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.609799 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce4092b7ccc4888b5cbe3b38bee2e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.610507 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09f5265d85c6496d85c555b2b847373a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.611021 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351cd60b076a49949f03640c981845c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.611515 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 848e414f6e134761818039f7e98500ac. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:29.613236 2053 leader_election.cc:343] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: ca4611c0a494425ca50d599b310872c8
I20251026 02:56:29.614571 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 09f5265d85c6496d85c555b2b847373a (DEFAULT_TABLE table=default.Table_76d502990a87496d9d6d2d163e712939 [id=fe6fe74d50ea4174bc20c04bc77f8215]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.615093 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09f5265d85c6496d85c555b2b847373a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.614573 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 351cd60b076a49949f03640c981845c4 (DEFAULT_TABLE table=default.Table_7ba0c9a1ed6e49f09387b3e13faee903 [id=f94002f27c954a41b8e666b012123a98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.618849 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351cd60b076a49949f03640c981845c4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:29.635334 2195 consensus_peers.cc:597] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78a624d839934d3089b64e397d4360af. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:29.636446 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1a2522801364693be0bda6fa45a697e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:29.638021 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1b6613742a24c21a1db6ebe858a846f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.637799 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86d8e1e21c5f440d8baf76c357d01ad2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:29.638217 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e62064fbdffb4954a3a1227efd6327ee" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:29.643565 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 027dd92fc5cf4c9e847de3e55e0dac1c (DEFAULT_TABLE table=default.Table_3f562a6edad942ad94cb473dddea9cd5 [id=f0b9b23703cc497e916ef082c403f00e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.646302 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 027dd92fc5cf4c9e847de3e55e0dac1c (DEFAULT_TABLE table=default.Table_3f562a6edad942ad94cb473dddea9cd5 [id=f0b9b23703cc497e916ef082c403f00e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.646521 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 3aed4584b49a4eccac2ba44454866d49 (DEFAULT_TABLE table=default.Table_173c6dd0f5b241c8a5ce1e918bde90a2 [id=8793c5ba1944467f937eff256b1d9832]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.646921 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 027dd92fc5cf4c9e847de3e55e0dac1c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:29.653558 2194 leader_election.cc:343] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.653739 2194 leader_election.cc:304] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:29.653950 2194 raft_consensus.cc:2686] T ca4611c0a494425ca50d599b310872c8 P 696476265495492790739ec126a08539: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251026 02:56:29.656275 2053 leader_election.cc:343] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:29.656334 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3af2695c41a4439bc4092b7cb1b2edf consensus state has no leader
I20251026 02:56:29.656428 2053 leader_election.cc:304] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:29.656989 2934 raft_consensus.cc:2749] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:29.658372 2053 leader_election.cc:343] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.658505 2053 leader_election.cc:304] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:29.658902 2934 raft_consensus.cc:2749] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:29.659919 2053 leader_election.cc:343] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.660048 2053 leader_election.cc:304] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:29.660279 2053 raft_consensus.cc:2686] T e62064fbdffb4954a3a1227efd6327ee P 0b6b0a6c4e144982b67980118cddba48: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251026 02:56:29.661449 2194 consensus_peers.cc:597] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:29.661799 2194 consensus_peers.cc:597] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:29.662899 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 3aed4584b49a4eccac2ba44454866d49 (DEFAULT_TABLE table=default.Table_173c6dd0f5b241c8a5ce1e918bde90a2 [id=8793c5ba1944467f937eff256b1d9832]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.663312 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aed4584b49a4eccac2ba44454866d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.674204 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5226d4516ce24ecda51dab35c86da132" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:29.682339 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca4611c0a494425ca50d599b310872c8" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:29.678534 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5226d4516ce24ecda51dab35c86da132" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:29.699306 2088 tablet_service.cc:1505] Processing CreateTablet for tablet a4f4d7f87916425999574fd5b9f2fdf7 (DEFAULT_TABLE table=default.Table_852e528c9c8c460d82d9e34569c8860d [id=b819d73c10f54c24a4f2939716b53f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.702064 2962 raft_consensus.cc:493] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:29.702239 2962 raft_consensus.cc:515] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.703034 2962 leader_election.cc:290] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:29.703190 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 (DEFAULT_TABLE table=default.Table_b5ccd4d8815f49b192cb9f1973042cce [id=d1ff1ff682064a5e96b158e9c64bdeb0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:29.704463 2195 leader_election.cc:343] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: a70d70abc3a34dac8cf9a747903fa78c
I20251026 02:56:29.708250 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_65854c69bc7b4af88efa140b222d480b" } modify_external_catalogs: true
I20251026 02:56:29.708498 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_65854c69bc7b4af88efa140b222d480b" } modify_external_catalogs: true
I20251026 02:56:29.710493 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce4092b7ccc4888b5cbe3b38bee2e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.716846 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 351cd60b076a49949f03640c981845c4 (DEFAULT_TABLE table=default.Table_7ba0c9a1ed6e49f09387b3e13faee903 [id=f94002f27c954a41b8e666b012123a98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.717420 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351cd60b076a49949f03640c981845c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.717760 2232 tablet_service.cc:1505] Processing CreateTablet for tablet a4f4d7f87916425999574fd5b9f2fdf7 (DEFAULT_TABLE table=default.Table_852e528c9c8c460d82d9e34569c8860d [id=b819d73c10f54c24a4f2939716b53f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.720672 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 09f5265d85c6496d85c555b2b847373a (DEFAULT_TABLE table=default.Table_76d502990a87496d9d6d2d163e712939 [id=fe6fe74d50ea4174bc20c04bc77f8215]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.721271 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09f5265d85c6496d85c555b2b847373a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.721702 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 8fe03778a1e842829cab25c90597a3cd (DEFAULT_TABLE table=default.Table_cf205f15299443d6a95fd58c1a38ee25 [id=3d04f0bd819b465caffc5fe105fa06b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.729233 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 027dd92fc5cf4c9e847de3e55e0dac1c (DEFAULT_TABLE table=default.Table_3f562a6edad942ad94cb473dddea9cd5 [id=f0b9b23703cc497e916ef082c403f00e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.729915 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 027dd92fc5cf4c9e847de3e55e0dac1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.731623 2962 raft_consensus.cc:493] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:29.731796 2962 raft_consensus.cc:515] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.732451 2962 leader_election.cc:290] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:29.733144 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "865fbd674db34f668fb5a2f5836a9b43" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:29.733364 2114 raft_consensus.cc:2468] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:29.733960 2195 leader_election.cc:304] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:29.734378 2962 raft_consensus.cc:2804] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:29.734464 2962 raft_consensus.cc:493] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:29.734549 2962 raft_consensus.cc:3060] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:29.748723 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4bd0ad4431c84593ae8581d943392a9d" } modify_external_catalogs: true
I20251026 02:56:29.748997 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4bd0ad4431c84593ae8581d943392a9d" } modify_external_catalogs: true
I20251026 02:56:29.754186 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 8fe03778a1e842829cab25c90597a3cd (DEFAULT_TABLE table=default.Table_cf205f15299443d6a95fd58c1a38ee25 [id=3d04f0bd819b465caffc5fe105fa06b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.754261 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 64415f5588db4df0a94676b650fb3631 (DEFAULT_TABLE table=default.Table_51c1281363e94fcaa024e7e1aca9b333 [id=78d00b5e85b6423ab7ada1d35df14f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.769445 1960 tablet_service.cc:1505] Processing CreateTablet for tablet a4f4d7f87916425999574fd5b9f2fdf7 (DEFAULT_TABLE table=default.Table_852e528c9c8c460d82d9e34569c8860d [id=b819d73c10f54c24a4f2939716b53f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.770321 1949 tablet_service.cc:1505] Processing CreateTablet for tablet 3aed4584b49a4eccac2ba44454866d49 (DEFAULT_TABLE table=default.Table_173c6dd0f5b241c8a5ce1e918bde90a2 [id=8793c5ba1944467f937eff256b1d9832]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.772356 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3f4de82346384f0faf6482156a0457d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:29.772487 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 8fe03778a1e842829cab25c90597a3cd (DEFAULT_TABLE table=default.Table_cf205f15299443d6a95fd58c1a38ee25 [id=3d04f0bd819b465caffc5fe105fa06b3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:29.773027 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f4de82346384f0faf6482156a0457d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.773237 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 64415f5588db4df0a94676b650fb3631 (DEFAULT_TABLE table=default.Table_51c1281363e94fcaa024e7e1aca9b333 [id=78d00b5e85b6423ab7ada1d35df14f9f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:29.773747 2053 leader_election.cc:343] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:29.774385 2053 leader_election.cc:343] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.774559 2053 leader_election.cc:304] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:29.774873 2934 raft_consensus.cc:2749] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:29.776968 2934 raft_consensus.cc:493] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:29.777108 2934 raft_consensus.cc:515] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.777755 2934 leader_election.cc:290] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:29.778672 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 64415f5588db4df0a94676b650fb3631 (DEFAULT_TABLE table=default.Table_51c1281363e94fcaa024e7e1aca9b333 [id=78d00b5e85b6423ab7ada1d35df14f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.778707 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a111848706b743c9beb2fbd21f59d7f3" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:29.779063 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 55d8d698ef0f4a81a0e269943fa41206 (DEFAULT_TABLE table=default.Table_79819a4391244437bf99313c2c857657 [id=ff60ae37981646cebc63eb64a073e2ff]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:29.779392 2053 leader_election.cc:343] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.785616 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 848e414f6e134761818039f7e98500ac (DEFAULT_TABLE table=default.Table_3107109fa3984828a1e2651e63ffb1ed [id=db65e2c421024deca4f6edc8ace95365]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.787500 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a70d70abc3a34dac8cf9a747903fa78c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:29.787814 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0e05eada8742471f87d73f5d960817c0
I20251026 02:56:29.788281 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "865fbd674db34f668fb5a2f5836a9b43" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:29.789606 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e05eada8742471f87d73f5d960817c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.790401 2968 tablet.cc:1722] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:29.791437 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e05eada8742471f87d73f5d960817c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:29.791790 2053 leader_election.cc:343] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.793020 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ffa4b05d8c84446297b336b55bbf5c6d" } new_table_name: "default.Table_43e64ec30d194fa89fa789b94321a269" modify_external_catalogs: true
I20251026 02:56:29.794879 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f4d7f87916425999574fd5b9f2fdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.795140 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe03778a1e842829cab25c90597a3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.799026 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64415f5588db4df0a94676b650fb3631. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.802261 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 79c18ebf56e045b6a52fae8871a9107f
I20251026 02:56:29.803560 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 79c18ebf56e045b6a52fae8871a9107f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.806530 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 55d8d698ef0f4a81a0e269943fa41206 (DEFAULT_TABLE table=default.Table_79819a4391244437bf99313c2c857657 [id=ff60ae37981646cebc63eb64a073e2ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.807098 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 79c18ebf56e045b6a52fae8871a9107f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:29.807169 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4098599e818f4dffbf00a6439bb66d9a" } new_table_name: "default.Table_5f40d3b6c1684d74a6a4577b9b3d0e85" modify_external_catalogs: true
I20251026 02:56:29.808696 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a111848706b743c9beb2fbd21f59d7f3" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:29.809753 2194 leader_election.cc:343] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.809906 2194 leader_election.cc:304] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:29.810182 2961 raft_consensus.cc:2749] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:29.810560 2194 leader_election.cc:343] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.811759 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 0e05eada8742471f87d73f5d960817c0 (DEFAULT_TABLE table=default.Table_b120712c89884acc8703a30980f5a265 [id=fe2535a7b72b47a683bcf2e7a94b6700]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:29.812239 2053 leader_election.cc:343] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.812492 2053 leader_election.cc:304] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:29.812851 2934 raft_consensus.cc:2749] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:29.817241 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 55d8d698ef0f4a81a0e269943fa41206 (DEFAULT_TABLE table=default.Table_79819a4391244437bf99313c2c857657 [id=ff60ae37981646cebc63eb64a073e2ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.818100 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e05eada8742471f87d73f5d960817c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.818603 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 79c18ebf56e045b6a52fae8871a9107f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.819254 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 0e05eada8742471f87d73f5d960817c0 (DEFAULT_TABLE table=default.Table_b120712c89884acc8703a30980f5a265 [id=fe2535a7b72b47a683bcf2e7a94b6700]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.821237 2226 tablet_service.cc:1505] Processing CreateTablet for tablet 0e05eada8742471f87d73f5d960817c0 (DEFAULT_TABLE table=default.Table_b120712c89884acc8703a30980f5a265 [id=fe2535a7b72b47a683bcf2e7a94b6700]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.821265 2092 tablet_service.cc:1505] Processing CreateTablet for tablet c1dd78f088f84a509c761c934c039016 (DEFAULT_TABLE table=default.Table_4bd0ad4431c84593ae8581d943392a9d [id=9283005d832543faa7a1de4696c33411]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.826004 1955 tablet_service.cc:1505] Processing CreateTablet for tablet c1dd78f088f84a509c761c934c039016 (DEFAULT_TABLE table=default.Table_4bd0ad4431c84593ae8581d943392a9d [id=9283005d832543faa7a1de4696c33411]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.828270 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet bfa68f2c942c43e2a02e9d333bdfa2a2
I20251026 02:56:29.828811 1656 catalog_manager.cc:6719] ReplaceTablet: tablet bfa68f2c942c43e2a02e9d333bdfa2a2 deleted and replaced by tablet 262f60fdb274476abf3533b4f47688b8
I20251026 02:56:29.829308 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet bfa68f2c942c43e2a02e9d333bdfa2a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 262f60fdb274476abf3533b4f47688b8) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:29.830076 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet bfa68f2c942c43e2a02e9d333bdfa2a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 262f60fdb274476abf3533b4f47688b8) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:29.830822 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet bfa68f2c942c43e2a02e9d333bdfa2a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 262f60fdb274476abf3533b4f47688b8) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:29.832906 2981 raft_consensus.cc:493] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:29.833199 2981 raft_consensus.cc:515] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.833707 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a4dc7d3d1de04432881624228345e98b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:29.834149 2981 leader_election.cc:290] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:29.834372 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4dc7d3d1de04432881624228345e98b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:29.834863 1916 leader_election.cc:343] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 7dec83bfb9e946db90c10d526af2dfca
I20251026 02:56:29.835647 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64415f5588db4df0a94676b650fb3631. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.835747 2372 ts_tablet_manager.cc:1929] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:29.835904 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e05eada8742471f87d73f5d960817c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.836336 2376 ts_tablet_manager.cc:1929] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:29.836464 2376 log.cc:1199] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/78a624d839934d3089b64e397d4360af
I20251026 02:56:29.836467 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1dd78f088f84a509c761c934c039016. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.836674 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55d8d698ef0f4a81a0e269943fa41206. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.836916 2376 ts_tablet_manager.cc:1950] T 78a624d839934d3089b64e397d4360af P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:29.836953 2962 raft_consensus.cc:515] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.837330 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f4d7f87916425999574fd5b9f2fdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.837651 2962 leader_election.cc:290] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:29.837836 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe03778a1e842829cab25c90597a3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.838284 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64415f5588db4df0a94676b650fb3631. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.838765 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55d8d698ef0f4a81a0e269943fa41206. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.839192 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e05eada8742471f87d73f5d960817c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.840557 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe03778a1e842829cab25c90597a3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.842242 1949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aed4584b49a4eccac2ba44454866d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.842603 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 348) took 2090 ms. Trace:
I20251026 02:56:29.842833 2376 rpcz_store.cc:276] 1026 02:56:27.751883 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.759022 (+ 7139us) service_pool.cc:225] Handling call
1026 02:56:29.187336 (+1428314us) ts_tablet_manager.cc:1151] Deleting tablet 78a624d839934d3089b64e397d4360af
1026 02:56:29.187345 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.698518 (+511173us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.836175 (+137657us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.842100 (+ 5925us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1428026}
I20251026 02:56:29.843488 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 349) took 2089 ms. Trace:
I20251026 02:56:29.843704 2376 rpcz_store.cc:276] 1026 02:56:27.753593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.759354 (+ 5761us) service_pool.cc:225] Handling call
1026 02:56:29.843331 (+2083977us) ts_tablet_manager.cc:1151] Deleting tablet f437d8b5f5784e8fbec10d0700e90348
1026 02:56:29.843342 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.843457 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2083703}
I20251026 02:56:29.844111 2376 tablet_replica.cc:333] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:29.844265 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 848e414f6e134761818039f7e98500ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.844465 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "865fbd674db34f668fb5a2f5836a9b43" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:29.844677 2114 raft_consensus.cc:3060] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:29.845919 2376 raft_consensus.cc:2243] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:29.845960 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f4d7f87916425999574fd5b9f2fdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.846115 2376 raft_consensus.cc:2272] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:29.846760 2376 ts_tablet_manager.cc:1916] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:29.848361 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 027dd92fc5cf4c9e847de3e55e0dac1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.849176 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 78a624d839934d3089b64e397d4360af (table default.Table_a72502f837fb4542b1f00bab5da37724 [id=4db9b61c88b5422fade5d42038bc5bcf]) successfully deleted
W20251026 02:56:29.849706 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet f437d8b5f5784e8fbec10d0700e90348 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f437d8b5f5784e8fbec10d0700e90348 already in progress: creating tablet
I20251026 02:56:29.850570 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "865fbd674db34f668fb5a2f5836a9b43" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:29.851593 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1dd78f088f84a509c761c934c039016. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.852072 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aed4584b49a4eccac2ba44454866d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.858294 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7dec83bfb9e946db90c10d526af2dfca" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:29.858577 2259 raft_consensus.cc:2468] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:29.859074 1915 leader_election.cc:304] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:29.864408 2381 tablet_bootstrap.cc:492] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:29.864727 2381 ts_tablet_manager.cc:1403] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.321s user 0.036s sys 0.024s
I20251026 02:56:29.865732 2381 raft_consensus.cc:359] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.866063 2381 raft_consensus.cc:385] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:29.866202 2381 raft_consensus.cc:740] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:29.866498 1660 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_016812daf097412081aff793ad75f72a [id=347d712aca4e477c89805846800fb824] alter complete (version 2)
I20251026 02:56:29.866636 2381 consensus_queue.cc:260] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:29.872021 2194 leader_election.cc:343] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:29.872496 2981 raft_consensus.cc:2804] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:29.873023 2234 tablet_service.cc:1505] Processing CreateTablet for tablet c1dd78f088f84a509c761c934c039016 (DEFAULT_TABLE table=default.Table_4bd0ad4431c84593ae8581d943392a9d [id=9283005d832543faa7a1de4696c33411]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:29.877512 2981 raft_consensus.cc:493] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:29.878669 2972 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 02:56:29.879050 2962 consensus_queue.cc:1048] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:29.880125 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55d8d698ef0f4a81a0e269943fa41206. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.881271 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e05eada8742471f87d73f5d960817c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.882273 2377 ts_tablet_manager.cc:1929] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:29.882459 2377 log.cc:1199] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7845ebd3e9ca4897b8fb19a48a010315
I20251026 02:56:29.882992 2377 ts_tablet_manager.cc:1950] T 7845ebd3e9ca4897b8fb19a48a010315 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:29.835886 2372 log.cc:1199] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/dbfe80d388644d0488cb2431a334c908
I20251026 02:56:29.883922 2372 ts_tablet_manager.cc:1950] T dbfe80d388644d0488cb2431a334c908 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:29.884439 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1dd78f088f84a509c761c934c039016. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:29.893292 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_62fd5a902d0648cea484f259d02de5f9" } modify_external_catalogs: true
I20251026 02:56:29.893538 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_62fd5a902d0648cea484f259d02de5f9" } modify_external_catalogs: true
I20251026 02:56:29.895071 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b0315e3c5d3f481b9db0d3570ad5a9c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:29.895645 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0315e3c5d3f481b9db0d3570ad5a9c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:29.896971 2114 raft_consensus.cc:2468] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:29.899992 2981 raft_consensus.cc:3060] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:29.904024 2195 leader_election.cc:304] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:29.912190 2962 raft_consensus.cc:2804] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:29.912489 2962 raft_consensus.cc:697] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:29.912832 2962 consensus_queue.cc:237] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.915110 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 338) took 2601 ms. Trace:
I20251026 02:56:29.915246 2377 rpcz_store.cc:276] 1026 02:56:27.313416 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.313482 (+ 66us) service_pool.cc:225] Handling call
1026 02:56:29.589212 (+2275730us) ts_tablet_manager.cc:1151] Deleting tablet 7845ebd3e9ca4897b8fb19a48a010315
1026 02:56:29.589222 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.799817 (+210595us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.882115 (+ 82298us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.915067 (+ 32952us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2271532}
I20251026 02:56:29.915767 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7845ebd3e9ca4897b8fb19a48a010315 (table default.Table_9c97bccbf4cb4d15a66dbb8130acb6d4 [id=10d722da66cd4661a1bb1ab82997dca8]) successfully deleted
I20251026 02:56:29.918402 2377 tablet_replica.cc:333] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:29.918753 2377 raft_consensus.cc:2243] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:29.919083 2377 raft_consensus.cc:2272] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:29.919842 2377 ts_tablet_manager.cc:1916] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:29.945609 2195 consensus_peers.cc:597] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:29.972131 2644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.61:37655, user_credentials={real_user=slave}} blocked reactor thread for 437036us
I20251026 02:56:29.985458 2378 tablet_bootstrap.cc:492] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:29.985949 2378 ts_tablet_manager.cc:1403] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.792s user 0.010s sys 0.007s
I20251026 02:56:29.987614 2378 raft_consensus.cc:359] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:29.992405 2962 consensus_queue.cc:1048] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251026 02:56:29.996578 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 435) took 1111 ms. Trace:
I20251026 02:56:29.999341 2372 rpcz_store.cc:276] 1026 02:56:28.884963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.885020 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:29.391029 (+506009us) ts_tablet_manager.cc:1151] Deleting tablet dbfe80d388644d0488cb2431a334c908
1026 02:56:29.391039 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:29.628643 (+237604us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.835568 (+206925us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:29.996523 (+160955us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":505777}
W20251026 02:56:30.002162 2194 consensus_peers.cc:597] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:30.002365 2378 raft_consensus.cc:385] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.002550 2378 raft_consensus.cc:740] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.003089 2378 consensus_queue.cc:260] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.005180 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet e89a98fcfdbb4011876b0ebc2af91f67
I20251026 02:56:30.005719 1662 catalog_manager.cc:6719] ReplaceTablet: tablet e89a98fcfdbb4011876b0ebc2af91f67 deleted and replaced by tablet d0f1d6967931482e9de40b0844da6315
I20251026 02:56:30.006546 2375 tablet_bootstrap.cc:492] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:30.006706 2375 ts_tablet_manager.cc:1403] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.418s user 0.021s sys 0.026s
I20251026 02:56:30.007644 2375 raft_consensus.cc:359] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.007853 2375 raft_consensus.cc:385] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.007937 2375 raft_consensus.cc:740] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.008168 2375 consensus_queue.cc:260] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.008819 2375 ts_tablet_manager.cc:1434] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:30.009133 2375 tablet_bootstrap.cc:492] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:30.009696 2377 ts_tablet_manager.cc:1929] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:30.010751 2377 log.cc:1199] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7dec83bfb9e946db90c10d526af2dfca
I20251026 02:56:30.011116 2375 tablet_bootstrap.cc:654] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.011186 2377 ts_tablet_manager.cc:1950] T 7dec83bfb9e946db90c10d526af2dfca P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:30.012053 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet e89a98fcfdbb4011876b0ebc2af91f67 with delete_type TABLET_DATA_DELETED (replaced by tablet d0f1d6967931482e9de40b0844da6315) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:30.012791 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet dbfe80d388644d0488cb2431a334c908 (table default.Table_ce7dea3ace314a5fa3ffeb9f16512725 [id=5964dceedb1c4d69b39051f9fe86b9dd]) successfully deleted
I20251026 02:56:30.014940 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet e89a98fcfdbb4011876b0ebc2af91f67 with delete_type TABLET_DATA_DELETED (replaced by tablet d0f1d6967931482e9de40b0844da6315) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:30.016609 1949 tablet_service.cc:1552] Processing DeleteTablet for tablet e89a98fcfdbb4011876b0ebc2af91f67 with delete_type TABLET_DATA_DELETED (replaced by tablet d0f1d6967931482e9de40b0844da6315) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.016830 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 433) took 1146 ms. Trace:
I20251026 02:56:30.017057 2381 ts_tablet_manager.cc:1434] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.152s user 0.002s sys 0.000s
I20251026 02:56:30.017206 2372 rpcz_store.cc:276] 1026 02:56:28.869951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.870380 (+ 429us) service_pool.cc:225] Handling call
1026 02:56:30.003250 (+1132870us) ts_tablet_manager.cc:1151] Deleting tablet 3b7d85a2941545ceaa16b22e8328b4ff
1026 02:56:30.003262 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.003379 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1116221}
W20251026 02:56:30.018963 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3b7d85a2941545ceaa16b22e8328b4ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b7d85a2941545ceaa16b22e8328b4ff already in progress: creating tablet
W20251026 02:56:30.019359 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet be1c0948c0ae4563b7cb644521914cb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be1c0948c0ae4563b7cb644521914cb9 already in progress: creating tablet
I20251026 02:56:30.019414 2372 tablet_replica.cc:333] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:30.019603 1662 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 358e805370ef410e9d65ee92a8d04f34 (table default.Table_71583d73a49142ae9fec31193fb53542 [id=e86d9fc892a8409590c555e25b27a12f]). Expected version 1 got 0
I20251026 02:56:30.019637 2372 raft_consensus.cc:2243] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:30.019836 2372 raft_consensus.cc:2272] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:30.020072 1662 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 865fbd674db34f668fb5a2f5836a9b43 (table default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa]). Expected version 1 got 0
I20251026 02:56:30.020817 2372 ts_tablet_manager.cc:1916] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:30.026666 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 95c9fbae62db4850b0f02a2443a3432b
I20251026 02:56:30.028772 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 95c9fbae62db4850b0f02a2443a3432b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:30.032269 2375 tablet_bootstrap.cc:492] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:30.032480 2375 ts_tablet_manager.cc:1403] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.006s
I20251026 02:56:30.033424 2375 raft_consensus.cc:359] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.033634 2375 raft_consensus.cc:385] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.033584 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 95c9fbae62db4850b0f02a2443a3432b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.033699 2375 raft_consensus.cc:740] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.033942 2375 consensus_queue.cc:260] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.034475 2375 ts_tablet_manager.cc:1434] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:30.034912 2375 tablet_bootstrap.cc:492] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:30.036758 2375 tablet_bootstrap.cc:654] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.040192 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_684ec52e10144760980fb35e3d5bad26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:30.040798 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_684ec52e10144760980fb35e3d5bad26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:30.055758 2375 tablet_bootstrap.cc:492] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:30.055975 2375 ts_tablet_manager.cc:1403] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.003s
I20251026 02:56:30.057065 2375 raft_consensus.cc:359] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.057348 2375 raft_consensus.cc:385] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.057468 2375 raft_consensus.cc:740] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.057757 2375 consensus_queue.cc:260] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.058418 2375 ts_tablet_manager.cc:1434] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:30.058722 2375 tablet_bootstrap.cc:492] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:30.060631 2375 tablet_bootstrap.cc:654] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.064867 2378 ts_tablet_manager.cc:1434] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.079s user 0.003s sys 0.000s
I20251026 02:56:30.066454 2378 tablet_bootstrap.cc:492] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:30.068763 2378 tablet_bootstrap.cc:654] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.073590 2381 tablet_bootstrap.cc:492] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:30.075416 2962 raft_consensus.cc:493] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.075618 2962 raft_consensus.cc:515] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:30.076110 1978 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.084s sys 0.093s
W20251026 02:56:30.076267 1978 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.537s user 0.084s sys 0.093s
I20251026 02:56:30.076941 2375 tablet_bootstrap.cc:492] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:30.077102 2375 ts_tablet_manager.cc:1403] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20251026 02:56:30.077925 2375 raft_consensus.cc:359] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.078135 2375 raft_consensus.cc:385] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.078204 2375 raft_consensus.cc:740] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.078451 2375 consensus_queue.cc:260] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.078928 2375 ts_tablet_manager.cc:1434] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:30.079128 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 95c9fbae62db4850b0f02a2443a3432b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:30.081614 2375 tablet_bootstrap.cc:492] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:30.083702 2375 tablet_bootstrap.cc:654] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:30.085114 2194 consensus_peers.cc:597] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:30.086097 2934 consensus_queue.cc:1048] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:30.089512 2962 leader_election.cc:290] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:30.090443 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "358e805370ef410e9d65ee92a8d04f34" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:30.091410 2194 leader_election.cc:343] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.092160 2375 tablet_bootstrap.cc:492] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:30.092339 2375 ts_tablet_manager.cc:1403] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20251026 02:56:30.092391 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "358e805370ef410e9d65ee92a8d04f34" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:30.092720 2119 raft_consensus.cc:2468] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:30.093171 2375 raft_consensus.cc:359] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.093371 2375 raft_consensus.cc:385] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.093452 2375 raft_consensus.cc:740] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.093400 2195 leader_election.cc:304] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:30.093660 2375 consensus_queue.cc:260] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.094185 2375 ts_tablet_manager.cc:1434] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:30.094678 2375 tablet_bootstrap.cc:492] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:30.095046 2962 raft_consensus.cc:2804] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:30.095196 2962 raft_consensus.cc:493] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:30.095309 2962 raft_consensus.cc:3060] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:30.096493 2375 tablet_bootstrap.cc:654] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.100947 2381 tablet_bootstrap.cc:654] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.103307 2381 tablet_bootstrap.cc:492] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:30.103549 2381 ts_tablet_manager.cc:1403] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.085s user 0.001s sys 0.004s
I20251026 02:56:30.104478 2381 raft_consensus.cc:359] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.104774 2381 raft_consensus.cc:385] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.104928 2381 raft_consensus.cc:740] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.105249 2381 consensus_queue.cc:260] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.107091 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 341) took 2793 ms. Trace:
I20251026 02:56:30.107254 2377 rpcz_store.cc:276] 1026 02:56:27.313765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.315479 (+ 1714us) service_pool.cc:225] Handling call
1026 02:56:29.918317 (+2602838us) ts_tablet_manager.cc:1151] Deleting tablet 7dec83bfb9e946db90c10d526af2dfca
1026 02:56:29.918331 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.004665 (+ 86334us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.009564 (+ 4899us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.107011 (+ 97447us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2599167}
I20251026 02:56:30.107724 2377 tablet_replica.cc:333] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:30.107980 2377 raft_consensus.cc:2243] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:30.112818 2934 raft_consensus.cc:493] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.113091 2934 raft_consensus.cc:515] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.114028 2934 leader_election.cc:290] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.114926 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0efdcc36ff5b47379e0a76512e05d379" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:30.108193 2377 raft_consensus.cc:2272] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:30.116286 2377 ts_tablet_manager.cc:1916] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:30.116381 2375 tablet_bootstrap.cc:492] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:30.116542 2375 ts_tablet_manager.cc:1403] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.009s
I20251026 02:56:30.117295 2375 raft_consensus.cc:359] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.117484 2375 raft_consensus.cc:385] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.117559 2375 raft_consensus.cc:740] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.117873 2375 consensus_queue.cc:260] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.118377 2375 ts_tablet_manager.cc:1434] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:30.118736 2375 tablet_bootstrap.cc:492] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:30.120576 2375 tablet_bootstrap.cc:654] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.121852 2381 ts_tablet_manager.cc:1434] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.018s user 0.003s sys 0.003s
I20251026 02:56:30.122202 2381 tablet_bootstrap.cc:492] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:30.123982 2381 tablet_bootstrap.cc:654] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:30.124260 2053 leader_election.cc:343] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.124480 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7dec83bfb9e946db90c10d526af2dfca (table default.Table_90929641047f40f1973c0e399816b5ef [id=327c67aeb10146e08119f7f3a7a11769]) successfully deleted
I20251026 02:56:30.124455 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0efdcc36ff5b47379e0a76512e05d379" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:30.134886 2053 leader_election.cc:343] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.135185 2053 leader_election.cc:304] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:30.136720 2934 raft_consensus.cc:2749] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:30.137748 2194 consensus_peers.cc:597] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:30.139925 2376 ts_tablet_manager.cc:1929] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:30.141072 2376 log.cc:1199] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/cfed3523205548508d1101586ee3e2a9
I20251026 02:56:30.142131 2376 ts_tablet_manager.cc:1950] T cfed3523205548508d1101586ee3e2a9 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:30.164995 2981 raft_consensus.cc:515] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.165953 2981 leader_election.cc:290] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:30.167073 1915 leader_election.cc:343] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 7dec83bfb9e946db90c10d526af2dfca
W20251026 02:56:30.167248 1916 leader_election.cc:343] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 7dec83bfb9e946db90c10d526af2dfca
I20251026 02:56:30.167395 1916 leader_election.cc:304] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:30.171217 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet c1dd78f088f84a509c761c934c039016
I20251026 02:56:30.172554 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet c1dd78f088f84a509c761c934c039016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:30.173264 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet c1dd78f088f84a509c761c934c039016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.174710 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2b60182354b04b6a83d7cc031c305d87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:30.175522 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b60182354b04b6a83d7cc031c305d87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:30.177603 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85f9fcae83a04c6e907c7e9c8cd95dbe consensus state has no leader
I20251026 02:56:30.178244 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1eaf8c8b19bf4a62bb69ffe0472d7910"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:30.178927 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1eaf8c8b19bf4a62bb69ffe0472d7910 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:30.180124 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db94738da3554fbda64c23fde3ef6c1b consensus state has no leader
I20251026 02:56:30.180505 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7c68f14075ce4e58bc9d8f08191f3f70" } new_table_name: "default.Table_3babf04294cc48ef92df6ffe8ab43e5a" modify_external_catalogs: true
I20251026 02:56:30.205441 2962 raft_consensus.cc:515] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.206192 2962 leader_election.cc:290] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:30.216516 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 352) took 2434 ms. Trace:
I20251026 02:56:30.216673 2376 rpcz_store.cc:276] 1026 02:56:27.781341 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.781412 (+ 71us) service_pool.cc:225] Handling call
1026 02:56:29.844044 (+2062632us) ts_tablet_manager.cc:1151] Deleting tablet cfed3523205548508d1101586ee3e2a9
1026 02:56:29.844056 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.012983 (+168927us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.139727 (+126744us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.215721 (+ 75994us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2062295}
I20251026 02:56:30.217186 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 359) took 2357 ms. Trace:
I20251026 02:56:30.217309 2376 rpcz_store.cc:276] 1026 02:56:27.859231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.859459 (+ 228us) service_pool.cc:225] Handling call
1026 02:56:30.217015 (+2357556us) ts_tablet_manager.cc:1151] Deleting tablet 9c7efb70c0804c149f36440a4a9e6ef8
1026 02:56:30.217025 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.217133 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2357303}
I20251026 02:56:30.217585 2376 tablet_replica.cc:333] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:30.217799 2376 raft_consensus.cc:2243] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:30.218012 2376 raft_consensus.cc:2272] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:30.218611 2376 ts_tablet_manager.cc:1916] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:30.220299 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "358e805370ef410e9d65ee92a8d04f34" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:30.221146 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet cfed3523205548508d1101586ee3e2a9 (table default.Table_e4fbc44faf984982986abfb227d6663d [id=b35a8189bbc04e94bb097c9e13a38ec6]) successfully deleted
W20251026 02:56:30.221534 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 9c7efb70c0804c149f36440a4a9e6ef8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c7efb70c0804c149f36440a4a9e6ef8 already in progress: creating tablet
W20251026 02:56:30.221995 2194 leader_election.cc:343] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.228077 2952 catalog_manager.cc:1269] Loaded cluster ID: 98a29211b5ce4bf98616dd4b8985c732
I20251026 02:56:30.228307 2952 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: loading cluster ID for follower catalog manager: success
I20251026 02:56:30.235585 2981 raft_consensus.cc:2749] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 02:56:30.238029 2952 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: acquiring CA information for follower catalog manager: success
I20251026 02:56:30.244871 2952 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 02:56:30.255918 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet c1dd78f088f84a509c761c934c039016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:30.257112 2377 ts_tablet_manager.cc:1929] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:30.257346 2377 log.cc:1199] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/5580ac28a98c4ec4a1d83fb9f9bf25ee
I20251026 02:56:30.257854 2377 ts_tablet_manager.cc:1950] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:30.258163 2375 tablet_bootstrap.cc:492] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:30.259194 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "358e805370ef410e9d65ee92a8d04f34" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:30.259456 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 99d318aa0d1847c58167bf0af9e75aa2
I20251026 02:56:30.259910 2375 ts_tablet_manager.cc:1403] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.141s user 0.007s sys 0.018s
I20251026 02:56:30.260134 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 1eabfc95caa84afc9a8956a7ac13fea6 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:30.259918 2119 raft_consensus.cc:3060] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:30.260844 2375 raft_consensus.cc:359] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.261070 2375 raft_consensus.cc:385] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.261204 2375 raft_consensus.cc:740] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.261503 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:30.261464 2375 consensus_queue.cc:260] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.263880 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 99d318aa0d1847c58167bf0af9e75aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:30.264132 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet a70d70abc3a34dac8cf9a747903fa78c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:30.264587 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 99d318aa0d1847c58167bf0af9e75aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:30.259177 1660 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_9fb0008346da44318ccc90cd69474750 [id=c7a6636ca94a4863a89fbef09c339bc1] alter complete (version 1)
W20251026 02:56:30.266472 1660 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.2:34337 (request call id 96) took 908 ms (client timeout 1000 ms). Trace:
I20251026 02:56:30.268932 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 86d8e1e21c5f440d8baf76c357d01ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:30.266656 1660 rpcz_store.cc:269] 1026 02:56:29.358506 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.373889 (+ 15383us) service_pool.cc:225] Handling call
1026 02:56:29.398052 (+ 24163us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x616000041254. stack: 00007f54638f59f6 00007f54638f57b0 00007f54628d13ad 0000000000983f04 0000000000983ecd 00007f5472bf36bd 00007f5472bf37b3 00007f546f7be441 00007f54774749dd 00007f54775fbaa8 00007f546e114ca6 00007f546e1149d5 00007f546a89127e 00007f546a89015e 00007f546a896395 00007f546a896cbf
1026 02:56:30.266437 (+868385us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:29.858362 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:29.858499 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:29.858506 (+ 7us) write_op.cc:435] Acquired schema lock
1026 02:56:29.858509 (+ 3us) tablet.cc:660] Decoding operations
1026 02:56:29.858636 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:29.858654 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:29.858658 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1026 02:56:29.858731 (+ 73us) tablet.cc:699] Row locks acquired
1026 02:56:29.858736 (+ 5us) write_op.cc:260] PREPARE: finished
1026 02:56:29.860961 (+ 2225us) write_op.cc:270] Start()
1026 02:56:29.861020 (+ 59us) write_op.cc:276] Timestamp: P: 1761447389860903 usec, L: 0
1026 02:56:29.861026 (+ 6us) op_driver.cc:348] REPLICATION: starting
1026 02:56:29.861260 (+ 234us) log.cc:844] Serialized 356 byte log entry
1026 02:56:29.865642 (+ 4382us) op_driver.cc:464] REPLICATION: finished
1026 02:56:29.865842 (+ 200us) write_op.cc:301] APPLY: starting
1026 02:56:29.865860 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:29.865907 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:29.865935 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:29.866008 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:29.866051 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:29.866061 (+ 10us) write_op.cc:312] APPLY: finished
1026 02:56:29.866166 (+ 105us) log.cc:844] Serialized 23 byte log entry
1026 02:56:29.866239 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:29.866262 (+ 23us) write_op.cc:454] Released schema lock
1026 02:56:29.866271 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:29.870692 (+ 4421us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:29.870699 (+ 7us) write_op.cc:454] Released schema lock
Related trace 'op':
1026 02:56:30.190216 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:30.190369 (+ 153us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:30.190378 (+ 9us) write_op.cc:435] Acquired schema lock
1026 02:56:30.190383 (+ 5us) tablet.cc:660] Decoding operations
1026 02:56:30.190515 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:30.190530 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:30.190537 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
1026 02:56:30.190616 (+ 79us) tablet.cc:699] Row locks acquired
1026 02:56:30.190719 (+ 103us) write_op.cc:260] PREPARE: finished
1026 02:56:30.190763 (+ 44us) write_op.cc:270] Start()
1026 02:56:30.190804 (+ 41us) write_op.cc:276] Timestamp: P: 1761447390190736 usec, L: 0
1026 02:56:30.190811 (+ 7us) op_driver.cc:348] REPLICATION: starting
1026 02:56:30.191034 (+ 223us) log.cc:844] Serialized 356 byte log entry
1026 02:56:30.255684 (+ 64650us) op_driver.cc:464] REPLICATION: finished
1026 02:56:30.256605 (+ 921us) write_op.cc:301] APPLY: starting
1026 02:56:30.256634 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:30.256694 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:30.256734 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:30.256815 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:30.256865 (+ 50us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:30.256877 (+ 12us) write_op.cc:312] APPLY: finished
1026 02:56:30.256997 (+ 120us) log.cc:844] Serialized 23 byte log entry
1026 02:56:30.257097 (+ 100us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:30.257126 (+ 29us) write_op.cc:454] Released schema lock
1026 02:56:30.257137 (+ 11us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:30.257235 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:30.257242 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6709,"reported_tablets":4,"spinlock_wait_cycles":53853056,"child_traces":[["op",{"apply.queue_time_us":143,"apply.run_cpu_time_us":501,"apply.run_wall_time_us":502,"num_ops":1,"prepare.queue_time_us":271166,"prepare.run_cpu_time_us":919,"prepare.run_wall_time_us":12323,"raft.queue_time_us":58,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":272,"replication_time_us":4584}],["op",{"apply.queue_time_us":841,"apply.run_cpu_time_us":622,"apply.run_wall_time_us":622,"num_ops":1,"prepare.queue_time_us":322735,"prepare.run_cpu_time_us":832,"prepare.run_wall_time_us":1308,"replication_time_us":64813}]]}
I20251026 02:56:30.273747 2381 tablet_bootstrap.cc:492] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:30.279863 2381 ts_tablet_manager.cc:1403] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.158s user 0.025s sys 0.068s
I20251026 02:56:30.282863 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_70e8c0a6af49412e870552049d11eb6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:30.283524 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70e8c0a6af49412e870552049d11eb6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:30.280957 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 340) took 2967 ms. Trace:
I20251026 02:56:30.283771 2377 rpcz_store.cc:276] 1026 02:56:27.313673 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.315889 (+ 2216us) service_pool.cc:225] Handling call
1026 02:56:30.107609 (+2791720us) ts_tablet_manager.cc:1151] Deleting tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee
1026 02:56:30.107623 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.206891 (+ 99268us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.256924 (+ 50033us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.280917 (+ 23993us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2788073}
I20251026 02:56:30.284279 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee (table default.Table_d2dcec579c684b92a92d98fb59ee3da1 [id=4f7edfb6d75c4a0b94ca99bb9cfe14f3]) successfully deleted
I20251026 02:56:30.284637 2377 tablet_replica.cc:333] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:30.284870 2377 raft_consensus.cc:2243] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:30.284988 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5cda8845fc8f42e9a04ed877e65b2be3" } new_table_name: "default.Table_a6b94cbe15a34811a6a613f0d749a414" modify_external_catalogs: true
I20251026 02:56:30.285207 2377 pending_rounds.cc:70] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:30.285300 2377 pending_rounds.cc:77] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888496604536832 op_type: NO_OP noop_request { }
I20251026 02:56:30.285413 2377 raft_consensus.cc:2889] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:30.285529 2377 raft_consensus.cc:2272] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:30.286118 2377 ts_tablet_manager.cc:1916] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:30.286553 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ecfe7bd401bb42f2bcdcbb956e55779d" } new_table_name: "default.Table_42b54262967149fda8be1e83c4579b5d" modify_external_catalogs: true
I20251026 02:56:30.288892 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4f13aec4f4284f51926349af4159512d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:30.289467 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f13aec4f4284f51926349af4159512d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:30.293062 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 with delete_type TABLET_DATA_DELETED (replaced by tablet 94886ad6d6bb45f39261725169774ac9) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.295382 2381 raft_consensus.cc:359] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.295639 2381 raft_consensus.cc:385] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.295708 2381 raft_consensus.cc:740] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.295989 2381 consensus_queue.cc:260] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.304277 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 99d318aa0d1847c58167bf0af9e75aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.305127 2375 ts_tablet_manager.cc:1434] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.045s user 0.008s sys 0.000s
I20251026 02:56:30.305531 2375 tablet_bootstrap.cc:492] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:30.307499 2375 tablet_bootstrap.cc:654] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.313123 2119 raft_consensus.cc:2468] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:30.313689 2372 ts_tablet_manager.cc:1929] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:30.313828 2372 log.cc:1199] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/9ec035b1caac450d9ab55e74a061ae38
I20251026 02:56:30.314245 2372 ts_tablet_manager.cc:1950] T 9ec035b1caac450d9ab55e74a061ae38 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:30.315135 2381 ts_tablet_manager.cc:1434] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
W20251026 02:56:30.315384 2053 consensus_peers.cc:597] T 47bfff40c181432896d6898ea50cd025 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:30.319336 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 8dbc12d49b094dbeadad2e76916ba9af from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:30.315071 2195 leader_election.cc:304] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:30.320269 2962 raft_consensus.cc:2804] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:30.320466 2962 raft_consensus.cc:697] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:30.320716 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_999c4a3cf18e418eaf40b7dbee0e5095"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:30.320740 2962 consensus_queue.cc:237] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:30.321362 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_999c4a3cf18e418eaf40b7dbee0e5095 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:30.325599 2378 tablet_bootstrap.cc:492] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:30.325781 2378 ts_tablet_manager.cc:1403] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.259s user 0.055s sys 0.069s
I20251026 02:56:30.326743 2378 raft_consensus.cc:359] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.326980 2378 raft_consensus.cc:385] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.327021 2381 tablet_bootstrap.cc:492] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:30.327080 2378 raft_consensus.cc:740] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.327342 2378 consensus_queue.cc:260] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.327976 2378 ts_tablet_manager.cc:1434] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:30.337105 2378 tablet_bootstrap.cc:492] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:30.339062 2378 tablet_bootstrap.cc:654] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.355329 2981 raft_consensus.cc:493] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.355549 2981 raft_consensus.cc:515] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.356585 2981 leader_election.cc:290] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:30.357033 1916 leader_election.cc:343] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 5580ac28a98c4ec4a1d83fb9f9bf25ee
W20251026 02:56:30.357213 1915 leader_election.cc:343] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 5580ac28a98c4ec4a1d83fb9f9bf25ee
I20251026 02:56:30.357352 1915 leader_election.cc:304] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:30.357519 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d1fa467b0a5a410aa8905dfabb5b5ea1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:30.358119 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1fa467b0a5a410aa8905dfabb5b5ea1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:30.358188 2981 raft_consensus.cc:2749] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:30.359175 2378 tablet_bootstrap.cc:492] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:30.359347 2378 ts_tablet_manager.cc:1403] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20251026 02:56:30.359750 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee with delete_type TABLET_DATA_DELETED (replaced by tablet 634d913c98c842a49d97a54f23752d26) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.360270 2378 raft_consensus.cc:359] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.360477 2378 raft_consensus.cc:385] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.360559 2378 raft_consensus.cc:740] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.360911 2378 consensus_queue.cc:260] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.361456 2378 ts_tablet_manager.cc:1434] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:30.361826 2378 tablet_bootstrap.cc:492] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:30.363626 2378 tablet_bootstrap.cc:654] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.366462 2381 tablet_bootstrap.cc:654] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.368398 2375 tablet_bootstrap.cc:492] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:30.368662 2375 ts_tablet_manager.cc:1403] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.000s
I20251026 02:56:30.369535 2375 raft_consensus.cc:359] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.369858 2375 raft_consensus.cc:385] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.370002 2375 raft_consensus.cc:740] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.370427 2375 consensus_queue.cc:260] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.371119 2375 ts_tablet_manager.cc:1434] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:30.375557 2375 tablet_bootstrap.cc:492] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:30.377456 2375 tablet_bootstrap.cc:654] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.384464 2378 tablet_bootstrap.cc:492] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:30.384708 2378 ts_tablet_manager.cc:1403] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.005s
I20251026 02:56:30.385864 2378 raft_consensus.cc:359] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.386222 2378 raft_consensus.cc:385] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.386344 2378 raft_consensus.cc:740] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.386709 2378 consensus_queue.cc:260] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.391084 2378 ts_tablet_manager.cc:1434] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
W20251026 02:56:30.398661 2195 consensus_peers.cc:597] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:30.401831 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 276bd3b97c97426daf802e1f57feb265 consensus state has no leader
I20251026 02:56:30.403712 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c9ed6201de674968b4f3b40267527312" } modify_external_catalogs: true
I20251026 02:56:30.403926 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c9ed6201de674968b4f3b40267527312" } modify_external_catalogs: true
I20251026 02:56:30.404204 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet ac19c5a847454a23ab16831198ff2a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:22 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.405855 2378 tablet_bootstrap.cc:492] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:30.407938 2378 tablet_bootstrap.cc:654] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.409677 2376 ts_tablet_manager.cc:1929] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:30.409811 2376 log.cc:1199] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/461412f3ad1e4186b1302cdf5a624ed4
I20251026 02:56:30.410230 2376 ts_tablet_manager.cc:1950] T 461412f3ad1e4186b1302cdf5a624ed4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:30.410732 2377 ts_tablet_manager.cc:1929] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:30.410856 2377 log.cc:1199] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/4a5343ba771a4e8ea2f467c4c619c1f9
I20251026 02:56:30.411303 2377 ts_tablet_manager.cc:1950] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:30.412597 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 363) took 2331 ms. Trace:
I20251026 02:56:30.412773 2376 rpcz_store.cc:276] 1026 02:56:28.081021 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.089473 (+ 8452us) service_pool.cc:225] Handling call
1026 02:56:30.217509 (+2128036us) ts_tablet_manager.cc:1151] Deleting tablet 461412f3ad1e4186b1302cdf5a624ed4
1026 02:56:30.217517 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.308579 (+ 91062us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.409525 (+100946us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.412537 (+ 3012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2127710}
I20251026 02:56:30.413550 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 364) took 2329 ms. Trace:
I20251026 02:56:30.413650 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 461412f3ad1e4186b1302cdf5a624ed4 (table default.Table_095bb281b8804e38b5bd56a1dbabacfa [id=6cab490620ef48f18ab808cd5d6c6af8]) successfully deleted
I20251026 02:56:30.413661 2376 rpcz_store.cc:276] 1026 02:56:28.083927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.089841 (+ 5914us) service_pool.cc:225] Handling call
1026 02:56:30.413371 (+2323530us) ts_tablet_manager.cc:1151] Deleting tablet 86e483c52e514f1da54335479f59d68e
1026 02:56:30.413382 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.413519 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2323238}
I20251026 02:56:30.414049 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 383) took 2040 ms. Trace:
I20251026 02:56:30.414151 2376 rpcz_store.cc:276] 1026 02:56:28.373867 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.373984 (+ 117us) service_pool.cc:225] Handling call
1026 02:56:30.413903 (+2039919us) ts_tablet_manager.cc:1151] Deleting tablet 122de8616b8f4fcf88971d5738c47e82
1026 02:56:30.413913 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.414021 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1991360}
I20251026 02:56:30.414479 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 385) took 2009 ms. Trace:
I20251026 02:56:30.414573 2376 rpcz_store.cc:276] 1026 02:56:28.404921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.405007 (+ 86us) service_pool.cc:225] Handling call
1026 02:56:30.414371 (+2009364us) ts_tablet_manager.cc:1151] Deleting tablet 17d2cc6b16d34331bb80c576c0cd2ce0
1026 02:56:30.414379 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.414452 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1981756}
I20251026 02:56:30.414877 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 386) took 1979 ms. Trace:
I20251026 02:56:30.414968 2376 rpcz_store.cc:276] 1026 02:56:28.435462 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.435521 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:30.414757 (+1979236us) ts_tablet_manager.cc:1151] Deleting tablet a53f25688e3b45dca68984e2c543d094
1026 02:56:30.414764 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.414836 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1974162}
I20251026 02:56:30.415282 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 387) took 1975 ms. Trace:
I20251026 02:56:30.415395 2376 rpcz_store.cc:276] 1026 02:56:28.439967 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.440035 (+ 68us) service_pool.cc:225] Handling call
1026 02:56:30.415163 (+1975128us) ts_tablet_manager.cc:1151] Deleting tablet 209fa71af13142a58efb9f3dab3eecd3
1026 02:56:30.415185 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.415255 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1970749}
I20251026 02:56:30.415727 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 388) took 1943 ms. Trace:
I20251026 02:56:30.415832 2376 rpcz_store.cc:276] 1026 02:56:28.472467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.472534 (+ 67us) service_pool.cc:225] Handling call
1026 02:56:30.415596 (+1943062us) ts_tablet_manager.cc:1151] Deleting tablet aa10cd990a944eab8d9eb66aaf20080c
1026 02:56:30.415605 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.415683 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1927167}
W20251026 02:56:30.418120 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 86e483c52e514f1da54335479f59d68e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86e483c52e514f1da54335479f59d68e already in progress: creating tablet
W20251026 02:56:30.418434 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 122de8616b8f4fcf88971d5738c47e82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 122de8616b8f4fcf88971d5738c47e82 already in progress: creating tablet
W20251026 02:56:30.418665 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 17d2cc6b16d34331bb80c576c0cd2ce0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17d2cc6b16d34331bb80c576c0cd2ce0 already in progress: creating tablet
W20251026 02:56:30.418876 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet a53f25688e3b45dca68984e2c543d094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a53f25688e3b45dca68984e2c543d094 already in progress: creating tablet
W20251026 02:56:30.419129 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 209fa71af13142a58efb9f3dab3eecd3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 209fa71af13142a58efb9f3dab3eecd3 already in progress: creating tablet
W20251026 02:56:30.419432 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet aa10cd990a944eab8d9eb66aaf20080c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa10cd990a944eab8d9eb66aaf20080c already in progress: creating tablet
I20251026 02:56:30.416186 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 389) took 1650 ms. Trace:
I20251026 02:56:30.420030 2376 rpcz_store.cc:276] 1026 02:56:28.766128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.766340 (+ 212us) service_pool.cc:225] Handling call
1026 02:56:30.416058 (+1649718us) ts_tablet_manager.cc:1151] Deleting tablet efd751ee4b1d433198dcd00c081f648d
1026 02:56:30.416067 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.416144 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1621987}
W20251026 02:56:30.420737 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet efd751ee4b1d433198dcd00c081f648d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efd751ee4b1d433198dcd00c081f648d already in progress: creating tablet
I20251026 02:56:30.420948 3014 raft_consensus.cc:493] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 209968db4dee47c49497a66c3ee750b7)
I20251026 02:56:30.421135 3014 raft_consensus.cc:515] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:30.421830 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 408) took 1583 ms. Trace:
I20251026 02:56:30.422041 3014 leader_election.cc:290] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 209968db4dee47c49497a66c3ee750b7 (127.1.150.62:44525), 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655)
I20251026 02:56:30.422230 2376 rpcz_store.cc:276] 1026 02:56:28.838567 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.839049 (+ 482us) service_pool.cc:225] Handling call
1026 02:56:30.421119 (+1582070us) ts_tablet_manager.cc:1151] Deleting tablet 41b7b0c36ba748f0a1565892dbe98a78
1026 02:56:30.421135 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.421784 (+ 649us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563535}
I20251026 02:56:30.422423 1673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 4 candidate_status { last_received { term: 3 index: 407 } } ignore_live_leader: false dest_uuid: "209968db4dee47c49497a66c3ee750b7" is_pre_election: true
I20251026 02:56:30.423321 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 410) took 1392 ms. Trace:
I20251026 02:56:30.423501 2376 rpcz_store.cc:276] 1026 02:56:29.030538 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.030612 (+ 74us) service_pool.cc:225] Handling call
1026 02:56:30.423187 (+1392575us) ts_tablet_manager.cc:1151] Deleting tablet dfd3e23d83a84648bd11cd258377aecb
1026 02:56:30.423200 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.423291 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1390561}
I20251026 02:56:30.424105 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 411) took 1393 ms. Trace:
I20251026 02:56:30.424301 2376 rpcz_store.cc:276] 1026 02:56:29.030630 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.032774 (+ 2144us) service_pool.cc:225] Handling call
1026 02:56:30.423927 (+1391153us) ts_tablet_manager.cc:1151] Deleting tablet dbfe80d388644d0488cb2431a334c908
1026 02:56:30.423937 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.424064 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1376313}
I20251026 02:56:30.424847 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 409) took 1394 ms. Trace:
I20251026 02:56:30.425017 2376 rpcz_store.cc:276] 1026 02:56:29.030378 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.031461 (+ 1083us) service_pool.cc:225] Handling call
1026 02:56:30.424714 (+1393253us) ts_tablet_manager.cc:1151] Deleting tablet 3b7d85a2941545ceaa16b22e8328b4ff
1026 02:56:30.424723 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.424805 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1374262}
I20251026 02:56:30.425532 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 412) took 1324 ms. Trace:
I20251026 02:56:30.425707 2376 rpcz_store.cc:276] 1026 02:56:29.101353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.101508 (+ 155us) service_pool.cc:225] Handling call
1026 02:56:30.425403 (+1323895us) ts_tablet_manager.cc:1151] Deleting tablet be1c0948c0ae4563b7cb644521914cb9
1026 02:56:30.425411 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.425503 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1323673}
I20251026 02:56:30.426203 2376 tablet_replica.cc:333] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:30.426486 2376 raft_consensus.cc:2243] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:30.426795 2376 raft_consensus.cc:2272] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:30.427525 2376 ts_tablet_manager.cc:1916] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:30.427525 2981 raft_consensus.cc:493] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.427805 2981 raft_consensus.cc:515] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.428624 2981 leader_election.cc:290] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.428887 1659 catalog_manager.cc:5649] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:30.429456 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 865fbd674db34f668fb5a2f5836a9b43 (table default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa]). Expected version 1 got 0
W20251026 02:56:30.429553 1916 leader_election.cc:343] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: c1b6257a3c8c4e1a8144ea45ff2a2a96
W20251026 02:56:30.429838 1915 leader_election.cc:343] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: c1b6257a3c8c4e1a8144ea45ff2a2a96
I20251026 02:56:30.429960 1915 leader_election.cc:304] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:30.430369 2981 raft_consensus.cc:2749] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:30.431221 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 22e4a433801a4cb58062d3e76c760b69 (table default.Table_f7c216f3ebd34c718ceef042a435b4cd [id=b4c0611288a949378eed16d32b5170b9]). Expected version 2 got 0
I20251026 02:56:30.432804 1744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 4 candidate_status { last_received { term: 3 index: 407 } } ignore_live_leader: false dest_uuid: "647c0de4ce394caca75179fe11658fc8" is_pre_election: true
I20251026 02:56:30.433622 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 339) took 3120 ms. Trace:
I20251026 02:56:30.433758 2377 rpcz_store.cc:276] 1026 02:56:27.313569 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.317773 (+ 4204us) service_pool.cc:225] Handling call
1026 02:56:30.284557 (+2966784us) ts_tablet_manager.cc:1151] Deleting tablet 4a5343ba771a4e8ea2f467c4c619c1f9
1026 02:56:30.284571 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.316163 (+ 31592us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.410582 (+ 94419us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.433566 (+ 22984us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2964814}
I20251026 02:56:30.434203 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 334) took 3133 ms. Trace:
I20251026 02:56:30.434304 2377 rpcz_store.cc:276] 1026 02:56:27.301228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.302133 (+ 905us) service_pool.cc:225] Handling call
1026 02:56:30.434062 (+3131929us) ts_tablet_manager.cc:1151] Deleting tablet ced7ad6a601d432cbc13b3199a0506e4
1026 02:56:30.434072 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.434174 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3106677}
I20251026 02:56:30.434571 2377 tablet_replica.cc:333] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:30.434788 2377 raft_consensus.cc:2243] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:30.435140 2377 raft_consensus.cc:2272] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:30.435734 2377 ts_tablet_manager.cc:1916] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:30.436672 2644 leader_election.cc:304] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 751b719b75ce4bc2a88a19091230ecb7; no voters: 209968db4dee47c49497a66c3ee750b7, 647c0de4ce394caca75179fe11658fc8
I20251026 02:56:30.436949 3014 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 02:56:30.437757 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04" } modify_external_catalogs: true
I20251026 02:56:30.437968 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04" } modify_external_catalogs: true
I20251026 02:56:30.440563 2378 tablet_bootstrap.cc:492] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:30.440734 2378 ts_tablet_manager.cc:1403] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.003s
I20251026 02:56:30.441674 2378 raft_consensus.cc:359] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.442271 2378 raft_consensus.cc:385] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.442360 2378 raft_consensus.cc:740] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.442711 2378 consensus_queue.cc:260] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.443662 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 4a5343ba771a4e8ea2f467c4c619c1f9 (table default.Table_7296727c5b2f4c95ac3ff62818bd84f2 [id=af7a0605122745588d028814d3583cc6]) successfully deleted
W20251026 02:56:30.443907 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ced7ad6a601d432cbc13b3199a0506e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ced7ad6a601d432cbc13b3199a0506e4 already in progress: creating tablet
I20251026 02:56:30.443876 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 4b49c0996c4d41cbaddcdf0b75846bcd (DEFAULT_TABLE table=default.Table_c9ed6201de674968b4f3b40267527312 [id=87a30e6347124e50b8ccb8f59c5b017a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:30.444149 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 41b7b0c36ba748f0a1565892dbe98a78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41b7b0c36ba748f0a1565892dbe98a78 already in progress: creating tablet
I20251026 02:56:30.444386 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b49c0996c4d41cbaddcdf0b75846bcd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:30.444368 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet dfd3e23d83a84648bd11cd258377aecb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfd3e23d83a84648bd11cd258377aecb already in progress: creating tablet
W20251026 02:56:30.444577 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet dbfe80d388644d0488cb2431a334c908 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbfe80d388644d0488cb2431a334c908 already in progress: creating tablet
W20251026 02:56:30.444761 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3b7d85a2941545ceaa16b22e8328b4ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b7d85a2941545ceaa16b22e8328b4ff already in progress: creating tablet
W20251026 02:56:30.444953 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet be1c0948c0ae4563b7cb644521914cb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be1c0948c0ae4563b7cb644521914cb9 already in progress: creating tablet
I20251026 02:56:30.446786 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 142043b5d558490c9399626c3764ff7e (DEFAULT_TABLE table=default.Table_69e73d553b1b47f6b4494421985d7889 [id=b4210c3d42a64ee9b1c853aa98da6f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.447009 2226 tablet_service.cc:1505] Processing CreateTablet for tablet 4b49c0996c4d41cbaddcdf0b75846bcd (DEFAULT_TABLE table=default.Table_c9ed6201de674968b4f3b40267527312 [id=87a30e6347124e50b8ccb8f59c5b017a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.447530 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 76c3e5fc281847f484ae71a19336a91c (DEFAULT_TABLE table=default.Table_500c2af7af964a57bc2b5c219aadaeb8 [id=0dfec0c4d1e64f78971bcc805ff7a7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.448575 1949 tablet_service.cc:1505] Processing CreateTablet for tablet 76c3e5fc281847f484ae71a19336a91c (DEFAULT_TABLE table=default.Table_500c2af7af964a57bc2b5c219aadaeb8 [id=0dfec0c4d1e64f78971bcc805ff7a7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.448987 1949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c3e5fc281847f484ae71a19336a91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.452560 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 3aa7a90a29ec430286786bf8aa9d4c02 (DEFAULT_TABLE table=default.Table_47d5f6b5a01b4d4d991fca2b89af298e [id=dd7f610dc1e74433a7c6d4f1a64a5fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.452984 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aa7a90a29ec430286786bf8aa9d4c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.454041 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 3aa7a90a29ec430286786bf8aa9d4c02 (DEFAULT_TABLE table=default.Table_47d5f6b5a01b4d4d991fca2b89af298e [id=dd7f610dc1e74433a7c6d4f1a64a5fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.455341 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 28dfe285139042349b5b9d6cbc59a778 (DEFAULT_TABLE table=default.Table_a72c8f4a62014bbeb2bc7cbb63991756 [id=e0c9a893670d406abf07988fc00abb29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.456360 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 00a8f0594509497a9cb856f7890da302 (DEFAULT_TABLE table=default.Table_f07a93acca3840ae8da80a0cc93ca97e [id=aeb3f03de7444d2f9f91d823c20cb16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.456789 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00a8f0594509497a9cb856f7890da302. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.457484 2231 tablet_service.cc:1505] Processing CreateTablet for tablet ded06ba1f80c4433ae78687e5990edce (DEFAULT_TABLE table=default.Table_a79373c41ad7448e903a6d939c57b345 [id=65f99e4eec4044b9b9a12835401e111e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:30.458523 1647 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 276bd3b97c97426daf802e1f57feb265 (table default.Table_3babf04294cc48ef92df6ffe8ab43e5a [id=e91358e10bab4589a6f4e815a85a5f7d], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 276bd3b97c97426daf802e1f57feb265 consensus state has no leader
I20251026 02:56:30.458703 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 1a1b8c440061429caea0f0803f110742 (DEFAULT_TABLE table=default.Table_5f363b68bc2c4e7685f25e4eefa532a8 [id=43cb7f538f824ae38059645230cc7e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.459017 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 142043b5d558490c9399626c3764ff7e (DEFAULT_TABLE table=default.Table_69e73d553b1b47f6b4494421985d7889 [id=b4210c3d42a64ee9b1c853aa98da6f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.459122 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1b8c440061429caea0f0803f110742. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.459403 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 142043b5d558490c9399626c3764ff7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.460799 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 28dfe285139042349b5b9d6cbc59a778 (DEFAULT_TABLE table=default.Table_a72c8f4a62014bbeb2bc7cbb63991756 [id=e0c9a893670d406abf07988fc00abb29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.461195 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28dfe285139042349b5b9d6cbc59a778. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.464602 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 00a8f0594509497a9cb856f7890da302 (DEFAULT_TABLE table=default.Table_f07a93acca3840ae8da80a0cc93ca97e [id=aeb3f03de7444d2f9f91d823c20cb16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.465225 1952 tablet_service.cc:1505] Processing CreateTablet for tablet ded06ba1f80c4433ae78687e5990edce (DEFAULT_TABLE table=default.Table_a79373c41ad7448e903a6d939c57b345 [id=65f99e4eec4044b9b9a12835401e111e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.465579 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded06ba1f80c4433ae78687e5990edce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.474543 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 1a1b8c440061429caea0f0803f110742 (DEFAULT_TABLE table=default.Table_5f363b68bc2c4e7685f25e4eefa532a8 [id=43cb7f538f824ae38059645230cc7e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.474872 1954 tablet_service.cc:1505] Processing CreateTablet for tablet e830f1aa1270420c9b28a0491a1a0960 (DEFAULT_TABLE table=default.Table_0ea3c4e29f264b34b9f6030b92262af6 [id=d33bf698f19d407da868aa876f64de5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.475134 2233 tablet_service.cc:1505] Processing CreateTablet for tablet e830f1aa1270420c9b28a0491a1a0960 (DEFAULT_TABLE table=default.Table_0ea3c4e29f264b34b9f6030b92262af6 [id=d33bf698f19d407da868aa876f64de5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.475292 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e830f1aa1270420c9b28a0491a1a0960. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.397101 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 437) took 1267 ms. Trace:
I20251026 02:56:30.475536 2372 rpcz_store.cc:276] 1026 02:56:29.129321 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.129384 (+ 63us) service_pool.cc:225] Handling call
1026 02:56:30.019344 (+889960us) ts_tablet_manager.cc:1151] Deleting tablet 9ec035b1caac450d9ab55e74a061ae38
1026 02:56:30.019356 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.145932 (+126576us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.313557 (+167625us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.397040 (+ 83483us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":889682}
I20251026 02:56:30.476089 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 440) took 1243 ms. Trace:
I20251026 02:56:30.476186 2372 rpcz_store.cc:276] 1026 02:56:29.232597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.232651 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:30.475858 (+1243207us) ts_tablet_manager.cc:1151] Deleting tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94
1026 02:56:30.475868 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.476031 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1242996}
I20251026 02:56:30.476570 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 442) took 1218 ms. Trace:
I20251026 02:56:30.476518 1960 tablet_service.cc:1505] Processing CreateTablet for tablet ceb7249968ff44c1b5be1d1d18605ffe (DEFAULT_TABLE table=default.Table_082149a0d40742629c17967fd62a789e [id=2b60132b7234412499d6f00f2b120d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.476653 2372 rpcz_store.cc:276] 1026 02:56:29.257837 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.258012 (+ 175us) service_pool.cc:225] Handling call
1026 02:56:30.476420 (+1218408us) ts_tablet_manager.cc:1151] Deleting tablet a08efd37968846d2958e54831488235c
1026 02:56:30.476429 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.476514 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1218194}
I20251026 02:56:30.476864 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceb7249968ff44c1b5be1d1d18605ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.476856 3006 raft_consensus.cc:493] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.477038 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 441) took 1224 ms. Trace:
I20251026 02:56:30.477134 2372 rpcz_store.cc:276] 1026 02:56:29.252997 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.253045 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:30.476880 (+1223835us) ts_tablet_manager.cc:1151] Deleting tablet b1ca5b8379c94deab1130bfe8bdc87b0
1026 02:56:30.476889 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.476966 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1212449}
I20251026 02:56:30.477475 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 443) took 1212 ms. Trace:
I20251026 02:56:30.477550 2372 rpcz_store.cc:276] 1026 02:56:29.264713 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.264800 (+ 87us) service_pool.cc:225] Handling call
1026 02:56:30.477340 (+1212540us) ts_tablet_manager.cc:1151] Deleting tablet dbf2a1ce78154415b23ab6d0efbc1f52
1026 02:56:30.477347 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.477421 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1212001}
I20251026 02:56:30.477877 2372 tablet_replica.cc:333] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:30.422816 2934 consensus_queue.cc:1048] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:30.433557 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 142043b5d558490c9399626c3764ff7e (DEFAULT_TABLE table=default.Table_69e73d553b1b47f6b4494421985d7889 [id=b4210c3d42a64ee9b1c853aa98da6f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.478574 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 142043b5d558490c9399626c3764ff7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.434057 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 76c3e5fc281847f484ae71a19336a91c (DEFAULT_TABLE table=default.Table_500c2af7af964a57bc2b5c219aadaeb8 [id=0dfec0c4d1e64f78971bcc805ff7a7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.482968 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c3e5fc281847f484ae71a19336a91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.434319 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 3aa7a90a29ec430286786bf8aa9d4c02 (DEFAULT_TABLE table=default.Table_47d5f6b5a01b4d4d991fca2b89af298e [id=dd7f610dc1e74433a7c6d4f1a64a5fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.486475 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aa7a90a29ec430286786bf8aa9d4c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.491807 2241 tablet_service.cc:1505] Processing CreateTablet for tablet ceb7249968ff44c1b5be1d1d18605ffe (DEFAULT_TABLE table=default.Table_082149a0d40742629c17967fd62a789e [id=2b60132b7234412499d6f00f2b120d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.433526 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 4b49c0996c4d41cbaddcdf0b75846bcd (DEFAULT_TABLE table=default.Table_c9ed6201de674968b4f3b40267527312 [id=87a30e6347124e50b8ccb8f59c5b017a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.492568 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b49c0996c4d41cbaddcdf0b75846bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.448902 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 28dfe285139042349b5b9d6cbc59a778 (DEFAULT_TABLE table=default.Table_a72c8f4a62014bbeb2bc7cbb63991756 [id=e0c9a893670d406abf07988fc00abb29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.496619 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28dfe285139042349b5b9d6cbc59a778. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.450865 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 00a8f0594509497a9cb856f7890da302 (DEFAULT_TABLE table=default.Table_f07a93acca3840ae8da80a0cc93ca97e [id=aeb3f03de7444d2f9f91d823c20cb16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.500885 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00a8f0594509497a9cb856f7890da302. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.500813 1954 tablet_service.cc:1505] Processing CreateTablet for tablet f51f8fef342b46abafff5bf12dba7184 (DEFAULT_TABLE table=default.Table_22ae79375e8e49d4b6d146d087ffa64b [id=4c170eaba0fd422f9fb0ce5e63e5c746]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.501366 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51f8fef342b46abafff5bf12dba7184. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:30.501801 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7834a85168364390a841fc883681c845 consensus state has no leader
I20251026 02:56:30.502204 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet a111848706b743c9beb2fbd21f59d7f3 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:30.452358 2093 tablet_service.cc:1505] Processing CreateTablet for tablet ded06ba1f80c4433ae78687e5990edce (DEFAULT_TABLE table=default.Table_a79373c41ad7448e903a6d939c57b345 [id=65f99e4eec4044b9b9a12835401e111e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.457422 3005 raft_consensus.cc:493] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.504400 3005 raft_consensus.cc:515] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.504547 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded06ba1f80c4433ae78687e5990edce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.505182 3005 leader_election.cc:290] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.464788 3007 raft_consensus.cc:493] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.505438 3007 raft_consensus.cc:515] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.506103 3007 leader_election.cc:290] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.470428 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 1a1b8c440061429caea0f0803f110742 (DEFAULT_TABLE table=default.Table_5f363b68bc2c4e7685f25e4eefa532a8 [id=43cb7f538f824ae38059645230cc7e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.506518 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1b8c440061429caea0f0803f110742. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.508661 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]) successfully deleted
I20251026 02:56:30.508811 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a5b68772a66402bbb0b7e00bf96fcc0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:30.508946 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 already in progress: creating tablet
W20251026 02:56:30.509152 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a08efd37968846d2958e54831488235c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a08efd37968846d2958e54831488235c already in progress: creating tablet
W20251026 02:56:30.509308 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet b1ca5b8379c94deab1130bfe8bdc87b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1ca5b8379c94deab1130bfe8bdc87b0 already in progress: creating tablet
W20251026 02:56:30.509470 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet dbf2a1ce78154415b23ab6d0efbc1f52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbf2a1ce78154415b23ab6d0efbc1f52 already in progress: creating tablet
I20251026 02:56:30.472604 2091 tablet_service.cc:1505] Processing CreateTablet for tablet e830f1aa1270420c9b28a0491a1a0960 (DEFAULT_TABLE table=default.Table_0ea3c4e29f264b34b9f6030b92262af6 [id=d33bf698f19d407da868aa876f64de5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.509865 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ced7ad6a601d432cbc13b3199a0506e4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:30.510110 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e830f1aa1270420c9b28a0491a1a0960. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.510937 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ced7ad6a601d432cbc13b3199a0506e4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:30.511277 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a5b68772a66402bbb0b7e00bf96fcc0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:30.511530 2053 leader_election.cc:343] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:30.511760 2053 leader_election.cc:343] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:30.512408 2053 leader_election.cc:343] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.512516 2053 leader_election.cc:304] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:30.512885 2053 leader_election.cc:343] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.512990 2053 leader_election.cc:304] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:30.474416 2085 tablet_service.cc:1505] Processing CreateTablet for tablet ceb7249968ff44c1b5be1d1d18605ffe (DEFAULT_TABLE table=default.Table_082149a0d40742629c17967fd62a789e [id=2b60132b7234412499d6f00f2b120d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.513926 2372 raft_consensus.cc:2243] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:30.514312 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceb7249968ff44c1b5be1d1d18605ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.514451 2372 raft_consensus.cc:2272] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:30.515061 2372 ts_tablet_manager.cc:1916] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:30.516001 2098 tablet_service.cc:1505] Processing CreateTablet for tablet f51f8fef342b46abafff5bf12dba7184 (DEFAULT_TABLE table=default.Table_22ae79375e8e49d4b6d146d087ffa64b [id=4c170eaba0fd422f9fb0ce5e63e5c746]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.516409 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51f8fef342b46abafff5bf12dba7184. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.516567 2227 tablet_service.cc:1505] Processing CreateTablet for tablet f51f8fef342b46abafff5bf12dba7184 (DEFAULT_TABLE table=default.Table_22ae79375e8e49d4b6d146d087ffa64b [id=4c170eaba0fd422f9fb0ce5e63e5c746]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.519924 3007 raft_consensus.cc:2749] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:30.520131 3007 raft_consensus.cc:2749] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:30.533025 2378 ts_tablet_manager.cc:1434] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.092s user 0.002s sys 0.000s
I20251026 02:56:30.537779 2381 tablet_bootstrap.cc:492] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:30.537988 2381 ts_tablet_manager.cc:1403] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.211s user 0.028s sys 0.028s
I20251026 02:56:30.538476 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f7c216f3ebd34c718ceef042a435b4cd" } modify_external_catalogs: true
I20251026 02:56:30.538682 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f7c216f3ebd34c718ceef042a435b4cd" } modify_external_catalogs: true
I20251026 02:56:30.538905 2381 raft_consensus.cc:359] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.539089 2381 raft_consensus.cc:385] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.539196 2381 raft_consensus.cc:740] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.539423 2381 consensus_queue.cc:260] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.539917 2381 ts_tablet_manager.cc:1434] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:30.539937 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 1406c22ee4ee4f94acecad9546208e79 (DEFAULT_TABLE table=default.Table_20e0d029dadf4384b1efbfbc372f4741 [id=ffe58c86529c46c2927c01d17db14c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.540201 2381 tablet_bootstrap.cc:492] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:30.540410 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1406c22ee4ee4f94acecad9546208e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.542850 2381 tablet_bootstrap.cc:654] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.543526 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b49c0996c4d41cbaddcdf0b75846bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.543737 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceb7249968ff44c1b5be1d1d18605ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.544015 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00a8f0594509497a9cb856f7890da302. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.544251 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aa7a90a29ec430286786bf8aa9d4c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.544797 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28dfe285139042349b5b9d6cbc59a778. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.547008 2378 tablet_bootstrap.cc:492] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:30.548377 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e830f1aa1270420c9b28a0491a1a0960. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.548831 2378 tablet_bootstrap.cc:654] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.552089 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51f8fef342b46abafff5bf12dba7184. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.553746 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded06ba1f80c4433ae78687e5990edce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.554711 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 142043b5d558490c9399626c3764ff7e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:30.555997 2053 consensus_peers.cc:597] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:30.477035 3006 raft_consensus.cc:515] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.564579 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 1406c22ee4ee4f94acecad9546208e79 (DEFAULT_TABLE table=default.Table_20e0d029dadf4384b1efbfbc372f4741 [id=ffe58c86529c46c2927c01d17db14c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.566859 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1b8c440061429caea0f0803f110742. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.567270 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1406c22ee4ee4f94acecad9546208e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.567595 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c3e5fc281847f484ae71a19336a91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.581665 2378 tablet_bootstrap.cc:492] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:30.581840 2378 ts_tablet_manager.cc:1403] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20251026 02:56:30.582657 2378 raft_consensus.cc:359] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.582850 2378 raft_consensus.cc:385] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.582937 2378 raft_consensus.cc:740] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.583146 2378 consensus_queue.cc:260] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.583729 2378 ts_tablet_manager.cc:1434] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251026 02:56:30.593087 1954 tablet_service.cc:1505] Processing CreateTablet for tablet e2103c245a374b87b5c8ebf89af64494 (DEFAULT_TABLE table=default.Table_73433427e5f34844932d213dfb63f8f9 [id=b12097f33b044be7a74b2dc5dc8d1271]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.593758 2378 tablet_bootstrap.cc:492] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:30.595662 2378 tablet_bootstrap.cc:654] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.598618 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 98fe44aab33b40eba819386c3bffca46 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:30.619104 2376 ts_tablet_manager.cc:1929] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:30.619295 2376 log.cc:1199] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/9ec035b1caac450d9ab55e74a061ae38
I20251026 02:56:30.619695 2376 ts_tablet_manager.cc:1950] T 9ec035b1caac450d9ab55e74a061ae38 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:30.620004 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2103c245a374b87b5c8ebf89af64494. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.628383 3006 leader_election.cc:290] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.630060 2377 ts_tablet_manager.cc:1929] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:30.630290 2377 log.cc:1199] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/81530bbf4a2d4e908add683475e594fb
I20251026 02:56:30.630779 2377 ts_tablet_manager.cc:1950] T 81530bbf4a2d4e908add683475e594fb P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:30.631512 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03da415f3a944a9a9679b3a45295ce35" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:30.632586 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03da415f3a944a9a9679b3a45295ce35" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:30.633805 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 262f60fdb274476abf3533b4f47688b8 (DEFAULT_TABLE table=default.Table_f51a5406d89743649dab37ef175a4666 [id=84eea78390b0408dbfeda88cafa9dcd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.634240 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262f60fdb274476abf3533b4f47688b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.638511 2381 tablet_bootstrap.cc:492] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:30.638708 2381 ts_tablet_manager.cc:1403] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.004s
I20251026 02:56:30.639622 2381 raft_consensus.cc:359] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.639837 2381 raft_consensus.cc:385] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.639926 2381 raft_consensus.cc:740] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.640336 2381 consensus_queue.cc:260] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.640897 2381 ts_tablet_manager.cc:1434] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:30.647024 2378 tablet_bootstrap.cc:492] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:30.647223 2378 ts_tablet_manager.cc:1403] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.013s
I20251026 02:56:30.647944 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 1406c22ee4ee4f94acecad9546208e79 (DEFAULT_TABLE table=default.Table_20e0d029dadf4384b1efbfbc372f4741 [id=ffe58c86529c46c2927c01d17db14c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.648164 2378 raft_consensus.cc:359] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.648422 2378 raft_consensus.cc:385] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251026 02:56:30.646206 2053 leader_election.cc:343] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.648499 2378 raft_consensus.cc:740] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.648598 2381 tablet_bootstrap.cc:492] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:30.648792 2378 consensus_queue.cc:260] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.649880 2378 ts_tablet_manager.cc:1434] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:30.650564 2381 tablet_bootstrap.cc:654] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.654893 2236 tablet_service.cc:1505] Processing CreateTablet for tablet e2103c245a374b87b5c8ebf89af64494 (DEFAULT_TABLE table=default.Table_73433427e5f34844932d213dfb63f8f9 [id=b12097f33b044be7a74b2dc5dc8d1271]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.658815 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 262f60fdb274476abf3533b4f47688b8 (DEFAULT_TABLE table=default.Table_f51a5406d89743649dab37ef175a4666 [id=84eea78390b0408dbfeda88cafa9dcd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.665539 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2103c245a374b87b5c8ebf89af64494. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.665755 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 262f60fdb274476abf3533b4f47688b8 (DEFAULT_TABLE table=default.Table_f51a5406d89743649dab37ef175a4666 [id=84eea78390b0408dbfeda88cafa9dcd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.666198 2375 tablet_bootstrap.cc:492] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:30.666534 2375 ts_tablet_manager.cc:1403] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.291s user 0.005s sys 0.000s
I20251026 02:56:30.668613 2375 raft_consensus.cc:359] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.669198 2375 raft_consensus.cc:385] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251026 02:56:30.669430 2053 leader_election.cc:343] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.670796 2375 raft_consensus.cc:740] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.671114 2375 consensus_queue.cc:260] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.673379 2053 leader_election.cc:304] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:30.673406 2375 ts_tablet_manager.cc:1434] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251026 02:56:30.674952 2998 raft_consensus.cc:493] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.675172 2934 raft_consensus.cc:2749] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:30.675122 2998 raft_consensus.cc:515] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.675841 2998 leader_election.cc:290] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:30.677565 2998 consensus_queue.cc:1048] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:30.678414 2375 tablet_bootstrap.cc:492] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:30.681252 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a111848706b743c9beb2fbd21f59d7f3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:30.681758 2118 raft_consensus.cc:2468] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:30.683239 2195 leader_election.cc:304] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:30.683343 2098 tablet_service.cc:1505] Processing CreateTablet for tablet e2103c245a374b87b5c8ebf89af64494 (DEFAULT_TABLE table=default.Table_73433427e5f34844932d213dfb63f8f9 [id=b12097f33b044be7a74b2dc5dc8d1271]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:30.683596 2998 raft_consensus.cc:2804] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:30.683686 2998 raft_consensus.cc:493] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:30.683791 2998 raft_consensus.cc:3060] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:30.687525 2381 tablet_bootstrap.cc:492] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:30.687700 2381 ts_tablet_manager.cc:1403] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.016s
I20251026 02:56:30.688585 2381 raft_consensus.cc:359] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.688776 2381 raft_consensus.cc:385] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.688871 2381 raft_consensus.cc:740] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.689106 2381 consensus_queue.cc:260] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.689720 2381 ts_tablet_manager.cc:1434] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:30.690264 2381 tablet_bootstrap.cc:492] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:30.692932 2375 tablet_bootstrap.cc:654] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.692198 2381 tablet_bootstrap.cc:654] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.698428 3024 tablet.cc:1722] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:30.704375 2934 raft_consensus.cc:493] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.704571 2934 raft_consensus.cc:515] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.705286 2934 leader_election.cc:290] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.706275 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "115ccccf16b74efc9f24b607cf0c74d1" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:30.706400 2934 raft_consensus.cc:493] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.706538 2934 raft_consensus.cc:515] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.706563 2259 raft_consensus.cc:2468] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:30.707162 2934 leader_election.cc:290] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.708081 2053 leader_election.cc:304] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:30.708987 2934 raft_consensus.cc:2804] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:30.708952 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2bfd9b55c484ef8a9c6345d56f794fb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:30.709111 2934 raft_consensus.cc:493] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:30.709197 2934 raft_consensus.cc:3060] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:30.712510 2053 leader_election.cc:343] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.717139 2381 tablet_bootstrap.cc:492] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:30.717339 2381 ts_tablet_manager.cc:1403] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.005s
I20251026 02:56:30.717943 2934 raft_consensus.cc:515] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.718205 2381 raft_consensus.cc:359] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.718447 2381 raft_consensus.cc:385] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.718396 2998 raft_consensus.cc:515] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.718526 2381 raft_consensus.cc:740] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.718766 2381 consensus_queue.cc:260] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.718858 2934 leader_election.cc:290] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.719105 2998 leader_election.cc:290] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:30.719919 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262f60fdb274476abf3533b4f47688b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.720000 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 336) took 3414 ms. Trace:
I20251026 02:56:30.720086 2377 rpcz_store.cc:276] 1026 02:56:27.305223 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.305532 (+ 309us) service_pool.cc:225] Handling call
1026 02:56:30.434518 (+3128986us) ts_tablet_manager.cc:1151] Deleting tablet 81530bbf4a2d4e908add683475e594fb
1026 02:56:30.434526 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.546664 (+112138us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.629894 (+ 83230us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.719936 (+ 90042us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":225,"tablet-delete.queue_time_us":3106798}
I20251026 02:56:30.720515 2377 tablet_replica.cc:333] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:30.720708 2377 raft_consensus.cc:2243] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:30.721031 2377 raft_consensus.cc:2272] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:30.721578 2377 ts_tablet_manager.cc:1916] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:30.725840 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a111848706b743c9beb2fbd21f59d7f3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:30.726148 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "115ccccf16b74efc9f24b607cf0c74d1" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:30.726433 2259 raft_consensus.cc:3060] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:30.726614 2114 raft_consensus.cc:3060] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:30.731056 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1406c22ee4ee4f94acecad9546208e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.731659 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2103c245a374b87b5c8ebf89af64494. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.737273 2381 ts_tablet_manager.cc:1434] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.020s user 0.003s sys 0.007s
I20251026 02:56:30.737670 2381 tablet_bootstrap.cc:492] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:30.739686 2381 tablet_bootstrap.cc:654] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.742259 3023 tablet.cc:1722] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:30.661017 2981 raft_consensus.cc:493] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.745719 2981 raft_consensus.cc:515] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.746621 2981 leader_election.cc:290] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.746817 2981 raft_consensus.cc:493] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.746948 2981 raft_consensus.cc:515] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.747076 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "340bf8c78a88413299b61a52f0a578f3" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:30.747539 2981 leader_election.cc:290] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.664737 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 413) took 1228 ms. Trace:
I20251026 02:56:30.747690 2376 rpcz_store.cc:276] 1026 02:56:29.436054 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.438589 (+ 2535us) service_pool.cc:225] Handling call
1026 02:56:30.426126 (+987537us) ts_tablet_manager.cc:1151] Deleting tablet 9ec035b1caac450d9ab55e74a061ae38
1026 02:56:30.426136 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.615417 (+189281us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.618960 (+ 3543us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.664666 (+ 45706us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":953154}
I20251026 02:56:30.673098 2378 tablet_bootstrap.cc:492] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:30.676774 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a111848706b743c9beb2fbd21f59d7f3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:30.706390 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "115ccccf16b74efc9f24b607cf0c74d1" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:30.748225 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 417) took 1278 ms. Trace:
I20251026 02:56:30.708693 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2bfd9b55c484ef8a9c6345d56f794fb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:30.748335 2376 rpcz_store.cc:276] 1026 02:56:29.470226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.470293 (+ 67us) service_pool.cc:225] Handling call
1026 02:56:30.748017 (+1277724us) ts_tablet_manager.cc:1151] Deleting tablet b1ca5b8379c94deab1130bfe8bdc87b0
1026 02:56:30.748030 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.748144 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1265798}
I20251026 02:56:30.719666 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "115ccccf16b74efc9f24b607cf0c74d1" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:30.723899 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a111848706b743c9beb2fbd21f59d7f3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:30.748700 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 418) took 1276 ms. Trace:
I20251026 02:56:30.748792 2376 rpcz_store.cc:276] 1026 02:56:29.472305 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.473012 (+ 707us) service_pool.cc:225] Handling call
1026 02:56:30.748542 (+1275530us) ts_tablet_manager.cc:1151] Deleting tablet a08efd37968846d2958e54831488235c
1026 02:56:30.748552 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.748638 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1241008}
I20251026 02:56:30.749142 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 416) took 1287 ms. Trace:
I20251026 02:56:30.749228 2376 rpcz_store.cc:276] 1026 02:56:29.462044 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.473965 (+ 11921us) service_pool.cc:225] Handling call
1026 02:56:30.748977 (+1275012us) ts_tablet_manager.cc:1151] Deleting tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94
1026 02:56:30.748985 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.749082 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1215569}
I20251026 02:56:30.749533 2376 tablet_replica.cc:333] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:30.749755 2376 raft_consensus.cc:2243] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:30.749943 2376 raft_consensus.cc:2272] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:30.750485 2376 ts_tablet_manager.cc:1916] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:30.750706 2378 tablet_bootstrap.cc:654] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.752135 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "340bf8c78a88413299b61a52f0a578f3" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:30.754439 1915 leader_election.cc:343] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 4a5343ba771a4e8ea2f467c4c619c1f9
I20251026 02:56:30.754700 1915 leader_election.cc:304] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:30.758059 2981 raft_consensus.cc:2749] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:30.758901 1916 leader_election.cc:343] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 4a5343ba771a4e8ea2f467c4c619c1f9
I20251026 02:56:30.759089 1916 leader_election.cc:304] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:30.759385 2981 raft_consensus.cc:2749] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:30.774394 2378 tablet_bootstrap.cc:492] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:30.774595 2378 ts_tablet_manager.cc:1403] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.102s user 0.007s sys 0.001s
I20251026 02:56:30.777204 2378 raft_consensus.cc:359] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.777415 2378 raft_consensus.cc:385] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.777525 2378 raft_consensus.cc:740] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.779681 2378 consensus_queue.cc:260] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.782658 2378 ts_tablet_manager.cc:1434] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251026 02:56:30.783038 2378 tablet_bootstrap.cc:492] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:30.787278 2378 tablet_bootstrap.cc:654] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.803699 2378 tablet_bootstrap.cc:492] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:30.803922 2378 ts_tablet_manager.cc:1403] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.005s
I20251026 02:56:30.804806 2378 raft_consensus.cc:359] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.805037 2378 raft_consensus.cc:385] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.805121 2378 raft_consensus.cc:740] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.807978 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1eabfc95caa84afc9a8956a7ac13fea6
I20251026 02:56:30.808998 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 1eabfc95caa84afc9a8956a7ac13fea6 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b8e1d79d2a247dba25540daf6b81bf0) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:30.809587 1661 catalog_manager.cc:6719] ReplaceTablet: tablet 1eabfc95caa84afc9a8956a7ac13fea6 deleted and replaced by tablet 5b8e1d79d2a247dba25540daf6b81bf0
I20251026 02:56:30.809794 1661 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39910 (request call id 553) took 1061 ms. Trace:
I20251026 02:56:30.810041 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 1eabfc95caa84afc9a8956a7ac13fea6 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b8e1d79d2a247dba25540daf6b81bf0) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:30.809988 1661 rpcz_store.cc:276] 1026 02:56:29.748276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.260021 (+511745us) service_pool.cc:225] Handling call
1026 02:56:30.288789 (+ 28768us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x616000041254. stack: 00007f54638f59f6 00007f54638f57b0 00007f54628d13ad 0000000000983f04 0000000000983ecd 00007f5472bf36bd 00007f5472bf37b3 00007f546f7be441 00007f54774749dd 00007f547760dfab 00007f546e124107 00007f546e123e35 00007f546a89127e 00007f546a89015e 00007f546a896395 00007f546a896cbf
1026 02:56:30.809759 (+520970us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:30.595904 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:30.596044 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:30.596051 (+ 7us) write_op.cc:435] Acquired schema lock
1026 02:56:30.596053 (+ 2us) tablet.cc:660] Decoding operations
1026 02:56:30.596159 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:30.596169 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:30.624328 (+ 28159us) tablet.cc:683] Acquiring locks for 2 operations
1026 02:56:30.624463 (+ 135us) tablet.cc:699] Row locks acquired
1026 02:56:30.624466 (+ 3us) write_op.cc:260] PREPARE: finished
1026 02:56:30.624519 (+ 53us) write_op.cc:270] Start()
1026 02:56:30.624565 (+ 46us) write_op.cc:276] Timestamp: P: 1761447390624503 usec, L: 0
1026 02:56:30.624569 (+ 4us) op_driver.cc:348] REPLICATION: starting
1026 02:56:30.624804 (+ 235us) log.cc:844] Serialized 622 byte log entry
1026 02:56:30.628798 (+ 3994us) op_driver.cc:464] REPLICATION: finished
1026 02:56:30.629439 (+ 641us) write_op.cc:301] APPLY: starting
1026 02:56:30.629458 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:30.629508 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:30.629538 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:30.629651 (+ 113us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:30.629720 (+ 69us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:30.629728 (+ 8us) write_op.cc:312] APPLY: finished
1026 02:56:30.629825 (+ 97us) log.cc:844] Serialized 29 byte log entry
1026 02:56:30.629898 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:30.629919 (+ 21us) write_op.cc:454] Released schema lock
1026 02:56:30.629926 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:30.630013 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:30.630017 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":65263488,"child_traces":[["op",{"apply.queue_time_us":572,"apply.run_cpu_time_us":561,"apply.run_wall_time_us":562,"num_ops":2,"prepare.queue_time_us":169485,"prepare.run_cpu_time_us":941,"prepare.run_wall_time_us":31386,"raft.queue_time_us":43,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":296,"replication_time_us":4199}]]}
I20251026 02:56:30.811051 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cf205f15299443d6a95fd58c1a38ee25" } modify_external_catalogs: true
I20251026 02:56:30.811300 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cf205f15299443d6a95fd58c1a38ee25" } modify_external_catalogs: true
I20251026 02:56:30.810688 2378 consensus_queue.cc:260] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.812801 2378 ts_tablet_manager.cc:1434] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20251026 02:56:30.813974 2378 tablet_bootstrap.cc:492] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:30.817718 2378 tablet_bootstrap.cc:654] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.818182 2114 raft_consensus.cc:2468] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:30.821938 2259 raft_consensus.cc:2468] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:30.822494 2372 ts_tablet_manager.cc:1929] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:30.822616 2372 log.cc:1199] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/9a08e8b7eb1949408f2151f2d7257e9d
I20251026 02:56:30.822710 2195 leader_election.cc:304] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:30.822939 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262f60fdb274476abf3533b4f47688b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:30.823068 2372 ts_tablet_manager.cc:1950] T 9a08e8b7eb1949408f2151f2d7257e9d P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:30.823164 2053 leader_election.cc:304] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:30.823498 2961 raft_consensus.cc:2804] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:30.823648 2961 raft_consensus.cc:697] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:30.823923 2961 consensus_queue.cc:237] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.825124 2934 raft_consensus.cc:2804] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:30.825282 2934 raft_consensus.cc:697] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:30.825591 2934 consensus_queue.cc:237] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.839216 2377 ts_tablet_manager.cc:1929] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:30.839495 2377 log.cc:1199] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/c570e5f9a93c45bda32ec92c68794714
I20251026 02:56:30.840025 2377 ts_tablet_manager.cc:1950] T c570e5f9a93c45bda32ec92c68794714 P 696476265495492790739ec126a08539: Deleting consensus metadata
W20251026 02:56:30.843879 2194 consensus_peers.cc:597] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:30.844756 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]) successfully deleted
I20251026 02:56:30.844945 2375 tablet_bootstrap.cc:492] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:30.845110 2375 ts_tablet_manager.cc:1403] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.167s user 0.013s sys 0.006s
W20251026 02:56:30.845664 2194 leader_election.cc:343] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.846100 2375 raft_consensus.cc:359] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.846388 2375 raft_consensus.cc:385] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.846477 2375 raft_consensus.cc:740] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
W20251026 02:56:30.846449 2053 leader_election.cc:343] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.846712 2375 consensus_queue.cc:260] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:30.847302 2053 leader_election.cc:343] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.847436 2053 leader_election.cc:304] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:30.847834 3006 raft_consensus.cc:2749] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:30.848471 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b1ca5b8379c94deab1130bfe8bdc87b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1ca5b8379c94deab1130bfe8bdc87b0 already in progress: creating tablet
W20251026 02:56:30.849097 2053 leader_election.cc:343] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:30.849840 2194 leader_election.cc:343] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:30.850558 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet a08efd37968846d2958e54831488235c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a08efd37968846d2958e54831488235c already in progress: creating tablet
W20251026 02:56:30.851641 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 already in progress: creating tablet
I20251026 02:56:30.852285 3030 raft_consensus.cc:493] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.852541 3030 raft_consensus.cc:515] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:30.854488 1915 leader_election.cc:343] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: ac19c5a847454a23ab16831198ff2a20
W20251026 02:56:30.854664 1916 leader_election.cc:343] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ac19c5a847454a23ab16831198ff2a20
I20251026 02:56:30.854694 3030 leader_election.cc:290] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.854791 1916 leader_election.cc:304] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:30.856683 1976 raft_consensus.cc:3060] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:30.858956 3006 raft_consensus.cc:493] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.859131 3006 raft_consensus.cc:515] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.859805 3006 leader_election.cc:290] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.860628 3030 raft_consensus.cc:2749] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:30.860687 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "548a7f9628ac4623b2c2f723f83c1e45" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:30.860934 2259 raft_consensus.cc:2468] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:30.861444 2053 leader_election.cc:304] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:30.862177 3006 raft_consensus.cc:2804] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:30.862322 3006 raft_consensus.cc:493] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:30.862412 3006 raft_consensus.cc:3060] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:30.862807 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 1eabfc95caa84afc9a8956a7ac13fea6 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b8e1d79d2a247dba25540daf6b81bf0) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.878577 1657 catalog_manager.cc:5649] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:30.879419 1657 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 548a7f9628ac4623b2c2f723f83c1e45 (table default.Table_f8baefeba2b54cb39affb2d4f1b0bef8 [id=d9a6b0b6f3e246a3aa8fbd1167205264]). Expected version 2 got 0
I20251026 02:56:30.879808 1657 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 22e4a433801a4cb58062d3e76c760b69 (table default.Table_f7c216f3ebd34c718ceef042a435b4cd [id=b4c0611288a949378eed16d32b5170b9]). Expected version 2 got 0
I20251026 02:56:30.879999 1657 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 865fbd674db34f668fb5a2f5836a9b43 (table default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa]). Expected version 1 got 0
I20251026 02:56:30.882790 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "548a7f9628ac4623b2c2f723f83c1e45" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:30.882877 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 340bf8c78a88413299b61a52f0a578f3 (table default.Table_1361fbf89aa04321a7876c1e43e27089 [id=c162ef89d2f14e91a24a6d8c2c37c261]). Expected version 4 got 0
W20251026 02:56:30.884178 2053 leader_election.cc:343] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.884094 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_171b77d4c8cb4fb48db2cb541d40d870"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:30.884738 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_171b77d4c8cb4fb48db2cb541d40d870 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:30.847353 2375 ts_tablet_manager.cc:1434] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:30.889865 2375 tablet_bootstrap.cc:492] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:30.894369 2375 tablet_bootstrap.cc:654] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.896745 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 335) took 3593 ms. Trace:
I20251026 02:56:30.896948 2377 rpcz_store.cc:276] 1026 02:56:27.303670 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.304754 (+ 1084us) service_pool.cc:225] Handling call
1026 02:56:30.720417 (+3415663us) ts_tablet_manager.cc:1151] Deleting tablet c570e5f9a93c45bda32ec92c68794714
1026 02:56:30.720426 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.818254 (+ 97828us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.839038 (+ 20784us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.896678 (+ 57640us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3390749}
I20251026 02:56:30.898499 2377 tablet_replica.cc:333] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:30.898808 2377 raft_consensus.cc:2243] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:30.899242 2377 pending_rounds.cc:70] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:30.899348 2377 pending_rounds.cc:77] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888499511422976 op_type: NO_OP noop_request { }
I20251026 02:56:30.899498 2377 raft_consensus.cc:2889] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:30.899648 2377 raft_consensus.cc:2272] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:30.900899 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 4b49c0996c4d41cbaddcdf0b75846bcd
I20251026 02:56:30.901305 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet c570e5f9a93c45bda32ec92c68794714 (table default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]) successfully deleted
I20251026 02:56:30.902132 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b49c0996c4d41cbaddcdf0b75846bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.902546 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7177931353164b2aa82c00b2807ccd63
I20251026 02:56:30.903980 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b49c0996c4d41cbaddcdf0b75846bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:30.904281 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 7177931353164b2aa82c00b2807ccd63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:30.904840 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b49c0996c4d41cbaddcdf0b75846bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:30.906489 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 7177931353164b2aa82c00b2807ccd63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:30.907239 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 7177931353164b2aa82c00b2807ccd63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.907348 2377 ts_tablet_manager.cc:1916] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:30.909439 2375 tablet_bootstrap.cc:492] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:30.909607 2375 ts_tablet_manager.cc:1403] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20251026 02:56:30.909986 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 155a23a93358487a9564bb5ac175b4eb with delete_type TABLET_DATA_DELETED (replaced by tablet 386e1bd4635b4a7e8e7f249b577fd978) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.910224 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.910245 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cbef830fee8242fcad0278eb16b8e127" } new_table_name: "default.Table_7667f4f99bf744579f1dc2b2d0f469eb" modify_external_catalogs: true
I20251026 02:56:30.910459 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 7845ebd3e9ca4897b8fb19a48a010315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.910504 2375 raft_consensus.cc:359] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.910737 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 8dbc12d49b094dbeadad2e76916ba9af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:30.917060 1657 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 358e805370ef410e9d65ee92a8d04f34 (table default.Table_71583d73a49142ae9fec31193fb53542 [id=e86d9fc892a8409590c555e25b27a12f]). Expected version 1 got 0
W20251026 02:56:30.917917 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5226d4516ce24ecda51dab35c86da132 consensus state has no leader
I20251026 02:56:30.918171 1656 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:39910 (request call id 552) took 1197 ms. Trace:
I20251026 02:56:30.918288 1656 rpcz_store.cc:276] 1026 02:56:29.720987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.237273 (+516286us) service_pool.cc:225] Handling call
1026 02:56:30.284915 (+ 47642us) spinlock_profiling.cc:243] Waited 47.4 ms on lock 0x616000041254. stack: 00007f54638f59f6 00007f54638f57b0 00007f54628d13ad 0000000000983f04 0000000000983ecd 00007f5472bf36bd 00007f5472bf37b3 00007f546f7be441 00007f54774749dd 00007f54776034f6 00007f546e11b146 00007f546e11ae75 00007f546a89127e 00007f546a89015e 00007f546a896395 00007f546a896cbf
1026 02:56:30.285180 (+ 265us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1026 02:56:30.285393 (+ 213us) catalog_manager.cc:3708] Apply alter schema
1026 02:56:30.285571 (+ 178us) catalog_manager.cc:3740] Acquired catalog manager lock
1026 02:56:30.285659 (+ 88us) catalog_manager.cc:3898] Updating metadata on disk
1026 02:56:30.442443 (+156784us) catalog_manager.cc:3929] Committing alterations to in-memory state
1026 02:56:30.918096 (+475653us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:30.407238 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:30.407334 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:30.407339 (+ 5us) write_op.cc:435] Acquired schema lock
1026 02:56:30.407342 (+ 3us) tablet.cc:660] Decoding operations
1026 02:56:30.407437 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:30.407445 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:30.407449 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1026 02:56:30.407496 (+ 47us) tablet.cc:699] Row locks acquired
1026 02:56:30.407499 (+ 3us) write_op.cc:260] PREPARE: finished
1026 02:56:30.407525 (+ 26us) write_op.cc:270] Start()
1026 02:56:30.407551 (+ 26us) write_op.cc:276] Timestamp: P: 1761447390407481 usec, L: 0
1026 02:56:30.407555 (+ 4us) op_driver.cc:348] REPLICATION: starting
1026 02:56:30.407685 (+ 130us) log.cc:844] Serialized 405 byte log entry
1026 02:56:30.434610 (+ 26925us) op_driver.cc:464] REPLICATION: finished
1026 02:56:30.441937 (+ 7327us) write_op.cc:301] APPLY: starting
1026 02:56:30.441957 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:30.442002 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:30.442040 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:30.442096 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:30.442128 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:30.442134 (+ 6us) write_op.cc:312] APPLY: finished
1026 02:56:30.442215 (+ 81us) log.cc:844] Serialized 23 byte log entry
1026 02:56:30.442286 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:30.442310 (+ 24us) write_op.cc:454] Released schema lock
1026 02:56:30.442319 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:30.493734 (+ 51415us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:30.493745 (+ 11us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":108951808,"child_traces":[["op",{"apply.queue_time_us":7255,"apply.run_cpu_time_us":452,"apply.run_wall_time_us":452,"num_ops":1,"prepare.queue_time_us":121127,"prepare.run_cpu_time_us":640,"prepare.run_wall_time_us":86458,"raft.queue_time_us":59,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":160,"replication_time_us":27025}]]}
I20251026 02:56:30.919548 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_60d710dc906a41308e183f47acdda437"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:30.921509 3028 raft_consensus.cc:493] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.921700 3028 raft_consensus.cc:515] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:30.922195 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60d710dc906a41308e183f47acdda437 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:30.922523 3028 leader_election.cc:290] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:30.923233 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1a2522801364693be0bda6fa45a697e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:30.923709 2194 leader_election.cc:343] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.915148 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 444) took 1647 ms. Trace:
I20251026 02:56:30.928121 2375 raft_consensus.cc:385] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.928295 2375 raft_consensus.cc:740] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.928190 2372 rpcz_store.cc:276] 1026 02:56:29.267420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.267475 (+ 55us) service_pool.cc:225] Handling call
1026 02:56:30.477768 (+1210293us) ts_tablet_manager.cc:1151] Deleting tablet 9a08e8b7eb1949408f2151f2d7257e9d
1026 02:56:30.477775 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.717798 (+240023us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.822348 (+104550us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.915093 (+ 92745us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35826,"tablet-delete.queue_time_us":1209950}
I20251026 02:56:30.928920 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 445) took 1603 ms. Trace:
I20251026 02:56:30.928957 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 9a08e8b7eb1949408f2151f2d7257e9d (table default.Table_5f363b68bc2c4e7685f25e4eefa532a8 [id=43cb7f538f824ae38059645230cc7e94]) successfully deleted
I20251026 02:56:30.929078 2372 rpcz_store.cc:276] 1026 02:56:29.325241 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.325319 (+ 78us) service_pool.cc:225] Handling call
1026 02:56:30.928717 (+1603398us) ts_tablet_manager.cc:1151] Deleting tablet ce922040298d4952af13652eeda7a234
1026 02:56:30.928731 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.928879 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602826}
I20251026 02:56:30.929473 2372 tablet_replica.cc:333] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:30.929627 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ce922040298d4952af13652eeda7a234 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce922040298d4952af13652eeda7a234 already in progress: creating tablet
I20251026 02:56:30.929792 2372 raft_consensus.cc:2243] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:30.930037 2372 raft_consensus.cc:2272] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:30.930155 2375 consensus_queue.cc:260] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.930668 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b709e5e9bb364edc939c1b743ca9009e" } new_table_name: "default.Table_19637f030638473d8f906e789a7cdb01" modify_external_catalogs: true
I20251026 02:56:30.931823 2372 ts_tablet_manager.cc:1916] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:30.923729 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1a2522801364693be0bda6fa45a697e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:30.932394 2375 ts_tablet_manager.cc:1434] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.023s user 0.000s sys 0.003s
W20251026 02:56:30.932862 2195 leader_election.cc:343] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.933087 2195 leader_election.cc:304] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:30.935474 2375 tablet_bootstrap.cc:492] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:30.937462 2375 tablet_bootstrap.cc:654] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:30.938812 3028 raft_consensus.cc:2749] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:30.939695 3006 raft_consensus.cc:515] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.942917 3006 leader_election.cc:290] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.943567 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "548a7f9628ac4623b2c2f723f83c1e45" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:30.943934 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8fe03778a1e842829cab25c90597a3cd
I20251026 02:56:30.943867 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "548a7f9628ac4623b2c2f723f83c1e45" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:30.944139 2259 raft_consensus.cc:3060] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:30.948906 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fe03778a1e842829cab25c90597a3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:30.944336 2053 leader_election.cc:343] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.948964 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fe03778a1e842829cab25c90597a3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:30.951838 2376 ts_tablet_manager.cc:1929] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:30.952368 2376 log.cc:1199] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/7dec83bfb9e946db90c10d526af2dfca
I20251026 02:56:30.954588 2981 raft_consensus.cc:493] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:30.954798 2981 raft_consensus.cc:515] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:30.955034 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7af610f8acba426b84a2c85d1d4f1c5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:30.955471 2981 leader_election.cc:290] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:30.955679 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7af610f8acba426b84a2c85d1d4f1c5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:30.956136 3006 raft_consensus.cc:493] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:30.956619 1916 leader_election.cc:343] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9a08e8b7eb1949408f2151f2d7257e9d
I20251026 02:56:30.957455 3006 raft_consensus.cc:515] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.958837 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d90e62ae86b149b291dc29cf7442c6bc" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:30.959563 2053 leader_election.cc:343] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.960167 3006 leader_election.cc:290] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:30.960538 2376 ts_tablet_manager.cc:1950] T 7dec83bfb9e946db90c10d526af2dfca P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:30.967077 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d90e62ae86b149b291dc29cf7442c6bc" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:30.966799 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a08e8b7eb1949408f2151f2d7257e9d" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:30.968168 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fe03778a1e842829cab25c90597a3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:30.968982 2053 leader_election.cc:343] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:30.968981 1915 leader_election.cc:304] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:30.969172 2053 leader_election.cc:304] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:30.969558 3030 raft_consensus.cc:2749] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:30.970324 3006 raft_consensus.cc:2749] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:30.971167 2381 tablet_bootstrap.cc:492] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:30.971350 2381 ts_tablet_manager.cc:1403] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.234s user 0.077s sys 0.070s
I20251026 02:56:30.972182 2381 raft_consensus.cc:359] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.972440 2381 raft_consensus.cc:385] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:30.972524 2381 raft_consensus.cc:740] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:30.972741 2381 consensus_queue.cc:260] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:30.973297 2381 ts_tablet_manager.cc:1434] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:30.973605 2381 tablet_bootstrap.cc:492] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:30.970397 3043 tablet.cc:1722] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:30.993537 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 98fe44aab33b40eba819386c3bffca46
I20251026 02:56:30.994058 1654 catalog_manager.cc:6719] ReplaceTablet: tablet 98fe44aab33b40eba819386c3bffca46 deleted and replaced by tablet 03aed9babfbe4acdad21d8ee22dbac1d
I20251026 02:56:30.994554 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 98fe44aab33b40eba819386c3bffca46 with delete_type TABLET_DATA_DELETED (replaced by tablet 03aed9babfbe4acdad21d8ee22dbac1d) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:30.995512 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d212170f056a44dfb15cfcb7f9fbae2f consensus state has no leader
I20251026 02:56:30.996779 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 98fe44aab33b40eba819386c3bffca46 with delete_type TABLET_DATA_DELETED (replaced by tablet 03aed9babfbe4acdad21d8ee22dbac1d) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.000337 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 98fe44aab33b40eba819386c3bffca46 with delete_type TABLET_DATA_DELETED (replaced by tablet 03aed9babfbe4acdad21d8ee22dbac1d) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:31.007365 2378 tablet_bootstrap.cc:492] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:31.007607 2378 ts_tablet_manager.cc:1403] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.194s user 0.014s sys 0.009s
I20251026 02:56:31.008356 2381 tablet_bootstrap.cc:654] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.008739 2378 raft_consensus.cc:359] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.009004 2378 raft_consensus.cc:385] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.009159 2378 raft_consensus.cc:740] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.009176 2961 raft_consensus.cc:493] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.009451 2961 raft_consensus.cc:515] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.009526 2378 consensus_queue.cc:260] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.011132 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86d8e1e21c5f440d8baf76c357d01ad2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:31.011381 2114 raft_consensus.cc:2468] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:31.014318 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 419) took 1536 ms. Trace:
I20251026 02:56:31.014447 2376 rpcz_store.cc:276] 1026 02:56:29.478187 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.478355 (+ 168us) service_pool.cc:225] Handling call
1026 02:56:30.749439 (+1271084us) ts_tablet_manager.cc:1151] Deleting tablet 7dec83bfb9e946db90c10d526af2dfca
1026 02:56:30.749449 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:30.822358 (+ 72909us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:30.951153 (+128795us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.014275 (+ 63122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1215421}
I20251026 02:56:31.011919 2195 leader_election.cc:304] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:31.019016 3028 raft_consensus.cc:2804] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:31.018970 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 7dec83bfb9e946db90c10d526af2dfca (table default.Table_90929641047f40f1973c0e399816b5ef [id=327c67aeb10146e08119f7f3a7a11769]) successfully deleted
I20251026 02:56:31.019115 3028 raft_consensus.cc:493] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:31.019217 3028 raft_consensus.cc:3060] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.013926 2259 raft_consensus.cc:2468] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:31.020622 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 423) took 1482 ms. Trace:
I20251026 02:56:31.020759 2376 rpcz_store.cc:276] 1026 02:56:29.538405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.540277 (+ 1872us) service_pool.cc:225] Handling call
1026 02:56:31.014697 (+1474420us) ts_tablet_manager.cc:1151] Deleting tablet f2d97084b5914350afbfda5c2a34bf9a
1026 02:56:31.020431 (+ 5734us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.020580 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474164}
I20251026 02:56:31.020826 2053 leader_election.cc:304] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.021158 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 422) took 1482 ms. Trace:
I20251026 02:56:31.021263 2376 rpcz_store.cc:276] 1026 02:56:29.538268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.551595 (+ 13327us) service_pool.cc:225] Handling call
1026 02:56:31.021018 (+1469423us) ts_tablet_manager.cc:1151] Deleting tablet ce922040298d4952af13652eeda7a234
1026 02:56:31.021040 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.021127 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1469188}
I20251026 02:56:31.021333 3006 raft_consensus.cc:2804] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:31.021456 3006 raft_consensus.cc:697] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:31.021523 2376 tablet_replica.cc:333] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:31.021724 2376 raft_consensus.cc:2243] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:31.021730 3006 consensus_queue.cc:237] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.021899 2376 raft_consensus.cc:2272] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:31.022473 2376 ts_tablet_manager.cc:1916] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:31.024338 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet f2d97084b5914350afbfda5c2a34bf9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2d97084b5914350afbfda5c2a34bf9a already in progress: creating tablet
W20251026 02:56:31.024631 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ce922040298d4952af13652eeda7a234 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce922040298d4952af13652eeda7a234 already in progress: creating tablet
I20251026 02:56:31.011279 2961 leader_election.cc:290] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:31.030058 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86d8e1e21c5f440d8baf76c357d01ad2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:31.030040 3024 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251026 02:56:31.030755 2194 leader_election.cc:343] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.032626 3023 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:31.051757 2378 ts_tablet_manager.cc:1434] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.044s user 0.000s sys 0.002s
I20251026 02:56:31.053088 2375 tablet_bootstrap.cc:492] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:31.053390 2375 ts_tablet_manager.cc:1403] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.121s user 0.007s sys 0.005s
I20251026 02:56:31.054384 2375 raft_consensus.cc:359] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.054683 2375 raft_consensus.cc:385] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.054808 2375 raft_consensus.cc:740] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.055176 2375 consensus_queue.cc:260] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.055801 2375 ts_tablet_manager.cc:1434] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:31.056185 2375 tablet_bootstrap.cc:492] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:31.060191 2378 tablet_bootstrap.cc:492] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:31.063728 2375 tablet_bootstrap.cc:654] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.064544 2378 tablet_bootstrap.cc:654] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.079686 2375 tablet_bootstrap.cc:492] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:31.079963 2375 ts_tablet_manager.cc:1403] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.006s
I20251026 02:56:31.082963 2378 tablet_bootstrap.cc:492] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:31.083237 2378 ts_tablet_manager.cc:1403] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.002s
I20251026 02:56:31.084540 2381 tablet_bootstrap.cc:492] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:31.084764 2381 ts_tablet_manager.cc:1403] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.111s user 0.002s sys 0.006s
I20251026 02:56:31.084928 2378 raft_consensus.cc:359] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.085223 2378 raft_consensus.cc:385] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.085419 2375 raft_consensus.cc:359] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.085693 2378 raft_consensus.cc:740] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.085760 2381 raft_consensus.cc:359] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.086090 2381 raft_consensus.cc:385] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.086024 2378 consensus_queue.cc:260] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.086261 2375 raft_consensus.cc:385] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.086251 2381 raft_consensus.cc:740] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.086385 2375 raft_consensus.cc:740] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.086673 2375 consensus_queue.cc:260] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.086674 2381 consensus_queue.cc:260] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.089397 2381 ts_tablet_manager.cc:1434] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:31.089780 2381 tablet_bootstrap.cc:492] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:31.091960 2381 tablet_bootstrap.cc:654] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:31.095273 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31b6e08597bd450eb1875201cf594a82 consensus state has no leader
I20251026 02:56:31.095573 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2b6ab2f608284fb1aec83b83ab7da74f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:31.095714 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e66bf47ad1824afeafd5c77400d56044"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:31.096189 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b6ab2f608284fb1aec83b83ab7da74f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:31.096297 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e66bf47ad1824afeafd5c77400d56044 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.095306 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2b60182354b04b6a83d7cc031c305d87" } modify_external_catalogs: true
I20251026 02:56:31.098284 1654 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 340bf8c78a88413299b61a52f0a578f3 (table default.Table_1361fbf89aa04321a7876c1e43e27089 [id=c162ef89d2f14e91a24a6d8c2c37c261]). Expected version 4 got 0
I20251026 02:56:31.098325 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2b60182354b04b6a83d7cc031c305d87" } modify_external_catalogs: true
I20251026 02:56:31.099138 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 741792cf96164695b03de3fb86b12602 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:31.099712 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet f437d8b5f5784e8fbec10d0700e90348 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:31.100200 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0fb1c918f03c49af90f8630333b5bab7" } new_table_name: "default.Table_7713a2385fb24bb7bfecfaa78176c128" modify_external_catalogs: true
I20251026 02:56:31.103335 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8ab273ba34e54a89910acbe0f5370372"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:31.104568 2378 ts_tablet_manager.cc:1434] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251026 02:56:31.104943 2375 ts_tablet_manager.cc:1434] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
W20251026 02:56:31.103909 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ab273ba34e54a89910acbe0f5370372 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.110451 2378 tablet_bootstrap.cc:492] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:31.112479 2378 tablet_bootstrap.cc:654] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.113405 2375 tablet_bootstrap.cc:492] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:31.113879 2372 ts_tablet_manager.cc:1929] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:31.114073 2372 log.cc:1199] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/b1a2522801364693be0bda6fa45a697e
I20251026 02:56:31.116127 2375 tablet_bootstrap.cc:654] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.118320 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 22e4a433801a4cb58062d3e76c760b69
W20251026 02:56:31.120101 1659 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.3:36919 (request call id 77) took 837 ms (client timeout 1000 ms). Trace:
W20251026 02:56:31.120239 1659 rpcz_store.cc:269] 1026 02:56:30.282643 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.288080 (+ 5437us) service_pool.cc:225] Handling call
1026 02:56:31.120040 (+831960us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:30.627342 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:30.627474 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:30.627482 (+ 8us) write_op.cc:435] Acquired schema lock
1026 02:56:30.627485 (+ 3us) tablet.cc:660] Decoding operations
1026 02:56:30.627590 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:30.627600 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:30.627604 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1026 02:56:30.627660 (+ 56us) tablet.cc:699] Row locks acquired
1026 02:56:30.627662 (+ 2us) write_op.cc:260] PREPARE: finished
1026 02:56:30.776807 (+149145us) write_op.cc:270] Start()
1026 02:56:30.776901 (+ 94us) write_op.cc:276] Timestamp: P: 1761447390627687 usec, L: 0
1026 02:56:30.776907 (+ 6us) op_driver.cc:348] REPLICATION: starting
1026 02:56:30.777175 (+ 268us) log.cc:844] Serialized 527 byte log entry
1026 02:56:30.873356 (+ 96181us) op_driver.cc:464] REPLICATION: finished
1026 02:56:30.873509 (+ 153us) write_op.cc:301] APPLY: starting
1026 02:56:30.873529 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:30.873577 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:30.873610 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:30.873677 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:30.873714 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:30.873721 (+ 7us) write_op.cc:312] APPLY: finished
1026 02:56:30.873807 (+ 86us) log.cc:844] Serialized 23 byte log entry
1026 02:56:30.873881 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:30.873906 (+ 25us) write_op.cc:454] Released schema lock
1026 02:56:30.873916 (+ 10us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:30.873997 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:30.874001 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":87,"apply.run_cpu_time_us":481,"apply.run_wall_time_us":480,"num_ops":1,"prepare.queue_time_us":195398,"prepare.run_cpu_time_us":902,"prepare.run_wall_time_us":151310,"raft.queue_time_us":74,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":318,"replication_time_us":96412}]]}
I20251026 02:56:31.119711 3028 raft_consensus.cc:515] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.122836 3028 leader_election.cc:290] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:31.122869 1657 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.2:34337 (request call id 97) took 786 ms (client timeout 1000 ms). Trace:
W20251026 02:56:31.122957 1657 rpcz_store.cc:269] 1026 02:56:30.336125 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.352521 (+ 16396us) service_pool.cc:225] Handling call
1026 02:56:31.122813 (+770292us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:30.977564 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:30.977655 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:30.977660 (+ 5us) write_op.cc:435] Acquired schema lock
1026 02:56:30.977662 (+ 2us) tablet.cc:660] Decoding operations
1026 02:56:30.977736 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:30.977744 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:30.977749 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1026 02:56:30.977791 (+ 42us) tablet.cc:699] Row locks acquired
1026 02:56:30.977793 (+ 2us) write_op.cc:260] PREPARE: finished
1026 02:56:30.977817 (+ 24us) write_op.cc:270] Start()
1026 02:56:30.977874 (+ 57us) write_op.cc:276] Timestamp: P: 1761447390977773 usec, L: 0
1026 02:56:30.977878 (+ 4us) op_driver.cc:348] REPLICATION: starting
1026 02:56:30.978014 (+ 136us) log.cc:844] Serialized 527 byte log entry
1026 02:56:30.991488 (+ 13474us) op_driver.cc:464] REPLICATION: finished
1026 02:56:30.992997 (+ 1509us) write_op.cc:301] APPLY: starting
1026 02:56:30.993006 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:30.993029 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:30.993048 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:30.993077 (+ 29us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:30.993097 (+ 20us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:30.993104 (+ 7us) write_op.cc:312] APPLY: finished
1026 02:56:30.993143 (+ 39us) log.cc:844] Serialized 23 byte log entry
1026 02:56:30.993194 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:30.993203 (+ 9us) write_op.cc:454] Released schema lock
1026 02:56:30.993208 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":1483,"apply.run_cpu_time_us":253,"apply.run_wall_time_us":253,"num_ops":1,"prepare.queue_time_us":55486,"prepare.run_cpu_time_us":515,"prepare.run_wall_time_us":515,"replication_time_us":13597,"thread_start_us":101802,"threads_started":1}]]}
I20251026 02:56:31.123773 2372 ts_tablet_manager.cc:1950] T b1a2522801364693be0bda6fa45a697e P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:31.124763 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 446) took 1776 ms. Trace:
I20251026 02:56:31.124881 2372 rpcz_store.cc:276] 1026 02:56:29.348390 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.348659 (+ 269us) service_pool.cc:225] Handling call
1026 02:56:30.929388 (+1580729us) ts_tablet_manager.cc:1151] Deleting tablet b1a2522801364693be0bda6fa45a697e
1026 02:56:30.929399 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.013950 (+ 84551us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.113734 (+ 99784us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.124693 (+ 10959us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565801}
I20251026 02:56:31.125010 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86d8e1e21c5f440d8baf76c357d01ad2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:31.125412 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 447) took 1752 ms. Trace:
I20251026 02:56:31.125545 2372 rpcz_store.cc:276] 1026 02:56:29.373397 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.374624 (+ 1227us) service_pool.cc:225] Handling call
1026 02:56:31.125235 (+1750611us) ts_tablet_manager.cc:1151] Deleting tablet f2d97084b5914350afbfda5c2a34bf9a
1026 02:56:31.125250 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.125358 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1750206}
I20251026 02:56:31.125707 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 122de8616b8f4fcf88971d5738c47e82 with delete_type TABLET_DATA_DELETED (replaced by tablet 81775055d98349f696cdab6171ffd58b) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.125954 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 458) took 1336 ms. Trace:
I20251026 02:56:31.126072 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a08e8b7eb1949408f2151f2d7257e9d with delete_type TABLET_DATA_DELETED (replaced by tablet 1a1b8c440061429caea0f0803f110742) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.126094 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86d8e1e21c5f440d8baf76c357d01ad2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:31.126281 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet ced7ad6a601d432cbc13b3199a0506e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.126329 2114 raft_consensus.cc:3060] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.125967 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 0efdcc36ff5b47379e0a76512e05d379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.126885 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 03da415f3a944a9a9679b3a45295ce35 with delete_type TABLET_DATA_DELETED (replaced by tablet a6f9ae327aa4401baf66a737eaa93cf6) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.127131 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet ca4611c0a494425ca50d599b310872c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.120302 2377 ts_tablet_manager.cc:1929] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:31.127379 2377 log.cc:1199] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/8dbc12d49b094dbeadad2e76916ba9af
I20251026 02:56:31.127720 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet b1a2522801364693be0bda6fa45a697e (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]) successfully deleted
I20251026 02:56:31.127813 2377 ts_tablet_manager.cc:1950] T 8dbc12d49b094dbeadad2e76916ba9af P 696476265495492790739ec126a08539: Deleting consensus metadata
W20251026 02:56:31.128538 2194 leader_election.cc:343] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:31.128621 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet f2d97084b5914350afbfda5c2a34bf9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2d97084b5914350afbfda5c2a34bf9a already in progress: creating tablet
I20251026 02:56:31.128818 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 22e4a433801a4cb58062d3e76c760b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.130095 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 22e4a433801a4cb58062d3e76c760b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:31.130558 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 81530bbf4a2d4e908add683475e594fb with delete_type TABLET_DATA_DELETED (replaced by tablet 95c9fbae62db4850b0f02a2443a3432b) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.130759 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet b1a2522801364693be0bda6fa45a697e with delete_type TABLET_DATA_DELETED (replaced by tablet ff242ef9541a42d793c4484190956da6) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.130954 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet c570e5f9a93c45bda32ec92c68794714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.131875 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.132081 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 22e4a433801a4cb58062d3e76c760b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.126096 2372 rpcz_store.cc:276] 1026 02:56:29.789446 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.789503 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:31.125781 (+1336278us) ts_tablet_manager.cc:1151] Deleting tablet 0e05eada8742471f87d73f5d960817c0
1026 02:56:31.125790 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.125892 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1336039}
I20251026 02:56:31.132486 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 459) took 1329 ms. Trace:
W20251026 02:56:31.132519 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0e05eada8742471f87d73f5d960817c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e05eada8742471f87d73f5d960817c0 already in progress: creating tablet
I20251026 02:56:31.132629 2372 rpcz_store.cc:276] 1026 02:56:29.803389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.803455 (+ 66us) service_pool.cc:225] Handling call
1026 02:56:31.132302 (+1328847us) ts_tablet_manager.cc:1151] Deleting tablet 79c18ebf56e045b6a52fae8871a9107f
1026 02:56:31.132316 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.132429 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1328570}
W20251026 02:56:31.134042 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 79c18ebf56e045b6a52fae8871a9107f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79c18ebf56e045b6a52fae8871a9107f already in progress: creating tablet
I20251026 02:56:31.134234 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 462) took 1304 ms. Trace:
I20251026 02:56:31.134416 2372 rpcz_store.cc:276] 1026 02:56:29.829850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.829964 (+ 114us) service_pool.cc:225] Handling call
1026 02:56:31.134053 (+1304089us) ts_tablet_manager.cc:1151] Deleting tablet bfa68f2c942c43e2a02e9d333bdfa2a2
1026 02:56:31.134067 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.134177 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1303838}
W20251026 02:56:31.134919 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet bfa68f2c942c43e2a02e9d333bdfa2a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfa68f2c942c43e2a02e9d333bdfa2a2 already in progress: creating tablet
I20251026 02:56:31.135089 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 463) took 1122 ms. Trace:
I20251026 02:56:31.135246 2372 rpcz_store.cc:276] 1026 02:56:30.012757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.014847 (+ 2090us) service_pool.cc:225] Handling call
1026 02:56:31.134917 (+1120070us) ts_tablet_manager.cc:1151] Deleting tablet e89a98fcfdbb4011876b0ebc2af91f67
1026 02:56:31.134925 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.135037 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1119726}
W20251026 02:56:31.135672 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet e89a98fcfdbb4011876b0ebc2af91f67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e89a98fcfdbb4011876b0ebc2af91f67 already in progress: creating tablet
I20251026 02:56:31.135825 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 464) took 1107 ms. Trace:
I20251026 02:56:31.135898 2378 tablet_bootstrap.cc:492] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:31.136057 2378 ts_tablet_manager.cc:1403] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.006s
I20251026 02:56:31.136025 2372 rpcz_store.cc:276] 1026 02:56:30.028614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.028672 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:31.135656 (+1106984us) ts_tablet_manager.cc:1151] Deleting tablet 95c9fbae62db4850b0f02a2443a3432b
1026 02:56:31.135664 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.135775 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106732}
W20251026 02:56:31.136564 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 95c9fbae62db4850b0f02a2443a3432b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95c9fbae62db4850b0f02a2443a3432b already in progress: creating tablet
W20251026 02:56:31.137070 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet c1dd78f088f84a509c761c934c039016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1dd78f088f84a509c761c934c039016 already in progress: creating tablet
I20251026 02:56:31.137018 2378 raft_consensus.cc:359] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.137262 2378 raft_consensus.cc:385] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.137362 2378 raft_consensus.cc:740] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
W20251026 02:56:31.137470 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 99d318aa0d1847c58167bf0af9e75aa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99d318aa0d1847c58167bf0af9e75aa2 already in progress: creating tablet
I20251026 02:56:31.137632 2378 consensus_queue.cc:260] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.138243 2378 ts_tablet_manager.cc:1434] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:31.138511 2378 tablet_bootstrap.cc:492] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:31.140471 2378 tablet_bootstrap.cc:654] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.142452 2375 tablet_bootstrap.cc:492] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:31.148725 2375 ts_tablet_manager.cc:1403] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.002s
I20251026 02:56:31.149991 2375 raft_consensus.cc:359] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.150279 2375 raft_consensus.cc:385] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.150375 2375 raft_consensus.cc:740] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.150700 2375 consensus_queue.cc:260] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.151479 2375 ts_tablet_manager.cc:1434] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 02:56:31.151726 2372 tablet_replica.cc:333] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:31.152354 2375 tablet_bootstrap.cc:492] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:31.155144 2375 tablet_bootstrap.cc:654] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.157627 2381 tablet_bootstrap.cc:492] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:31.159967 2381 ts_tablet_manager.cc:1403] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.003s
I20251026 02:56:31.161033 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_000a2b7754d04bd197fca93eab72c154" } new_table_name: "default.Table_42d9349693d44bc5bff21ffa159fd32f" modify_external_catalogs: true
I20251026 02:56:31.162652 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8cfa87d9be80442b941db99ddc6021d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:31.163228 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cfa87d9be80442b941db99ddc6021d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.164511 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_53d4276e57e646faa0ed4bfab8ae8e82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:31.164978 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53d4276e57e646faa0ed4bfab8ae8e82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.167145 1661 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 903e65fa8c214dc9a50cb97cab101d22 (table default.Table_862bd13f099645e8aafd987e4e850ae6 [id=65b5770c4b9243b892bfab963a439bb9]). Expected version 3 got 0
I20251026 02:56:31.171968 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet c570e5f9a93c45bda32ec92c68794714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:31.181116 2381 raft_consensus.cc:359] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.181452 2381 raft_consensus.cc:385] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.181614 2381 raft_consensus.cc:740] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.181846 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a111848706b743c9beb2fbd21f59d7f3
I20251026 02:56:31.182390 1663 catalog_manager.cc:6719] ReplaceTablet: tablet a111848706b743c9beb2fbd21f59d7f3 deleted and replaced by tablet d7313a6396c541d7ac6d8ca2e219af90
I20251026 02:56:31.182618 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet a111848706b743c9beb2fbd21f59d7f3 with delete_type TABLET_DATA_DELETED (replaced by tablet d7313a6396c541d7ac6d8ca2e219af90) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:31.183641 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet a111848706b743c9beb2fbd21f59d7f3 with delete_type TABLET_DATA_DELETED (replaced by tablet d7313a6396c541d7ac6d8ca2e219af90) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.181957 2381 consensus_queue.cc:260] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.185906 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet a111848706b743c9beb2fbd21f59d7f3 with delete_type TABLET_DATA_DELETED (replaced by tablet d7313a6396c541d7ac6d8ca2e219af90) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.187103 2375 tablet_bootstrap.cc:492] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:31.187333 2375 ts_tablet_manager.cc:1403] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.000s
I20251026 02:56:31.188362 2381 ts_tablet_manager.cc:1434] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251026 02:56:31.188670 2381 tablet_bootstrap.cc:492] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:31.190438 2381 tablet_bootstrap.cc:654] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.197500 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_857acd1afff5412b8945e40b98c1dde6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:31.198243 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_857acd1afff5412b8945e40b98c1dde6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.198446 1658 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 903e65fa8c214dc9a50cb97cab101d22 (table default.Table_862bd13f099645e8aafd987e4e850ae6 [id=65b5770c4b9243b892bfab963a439bb9]). Expected version 3 got 0
I20251026 02:56:31.198673 1658 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet a821ed8a686c452a975acb96b5c74778 (table default.Table_016812daf097412081aff793ad75f72a [id=347d712aca4e477c89805846800fb824]). Expected version 2 got 0
I20251026 02:56:31.199515 1655 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet d856bfe2c84f4495ac4489567a084815
I20251026 02:56:31.199863 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e4a6aa2d2b9f4653b73ae643cfe689cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:31.200008 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 4b5e61afe90e4bc4a2897de21e0b3919 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:31.200448 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4a6aa2d2b9f4653b73ae643cfe689cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:31.201798 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a3f3fc3c7a04d05bae28472251f3acc consensus state has no leader
I20251026 02:56:31.202165 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3babf04294cc48ef92df6ffe8ab43e5a" } modify_external_catalogs: true
I20251026 02:56:31.202390 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3babf04294cc48ef92df6ffe8ab43e5a" } modify_external_catalogs: true
I20251026 02:56:31.203104 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3d25c519f16647ca962f3862c6cd4f98" } modify_external_catalogs: true
I20251026 02:56:31.203233 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3d25c519f16647ca962f3862c6cd4f98" } modify_external_catalogs: true
W20251026 02:56:31.203747 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87895fb88c284a4f8d7c58a5ee48ba8e consensus state has no leader
I20251026 02:56:31.204105 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_862bd13f099645e8aafd987e4e850ae6" } new_table_name: "default.Table_2d168c00cb884360bdb2475bd391f576" modify_external_catalogs: true
I20251026 02:56:31.212273 2375 raft_consensus.cc:359] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.212584 2375 raft_consensus.cc:385] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.212699 2375 raft_consensus.cc:740] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.213372 2114 raft_consensus.cc:2468] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:31.213544 2376 ts_tablet_manager.cc:1929] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:31.213943 2195 leader_election.cc:304] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.214504 2961 raft_consensus.cc:2804] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:31.214679 2961 raft_consensus.cc:697] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:31.214972 2961 consensus_queue.cc:237] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.216607 2372 raft_consensus.cc:2243] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:31.217024 2372 raft_consensus.cc:2272] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:31.217664 2372 ts_tablet_manager.cc:1916] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:31.219470 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 337) took 3909 ms. Trace:
I20251026 02:56:31.219606 2377 rpcz_store.cc:276] 1026 02:56:27.309695 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.312453 (+ 2758us) service_pool.cc:225] Handling call
1026 02:56:30.898372 (+3585919us) ts_tablet_manager.cc:1151] Deleting tablet 8dbc12d49b094dbeadad2e76916ba9af
1026 02:56:30.898397 (+ 25us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.017728 (+119331us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.120131 (+102403us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.219420 (+ 99289us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3566508}
I20251026 02:56:31.220062 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 348) took 3726 ms. Trace:
I20251026 02:56:31.220170 2377 rpcz_store.cc:276] 1026 02:56:27.493582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.493630 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:31.219918 (+3726288us) ts_tablet_manager.cc:1151] Deleting tablet 2524ee0c964345459c3e62f1e39b8f9e
1026 02:56:31.219929 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.220026 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3726055}
I20251026 02:56:31.220553 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 353) took 3691 ms. Trace:
I20251026 02:56:31.220645 2377 rpcz_store.cc:276] 1026 02:56:27.528816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.528859 (+ 43us) service_pool.cc:225] Handling call
1026 02:56:31.220417 (+3691558us) ts_tablet_manager.cc:1151] Deleting tablet ec13c5386abe414ba25589e03d5b850a
1026 02:56:31.220427 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.220526 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3691094}
I20251026 02:56:31.220945 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 354) took 3691 ms. Trace:
I20251026 02:56:31.221038 2377 rpcz_store.cc:276] 1026 02:56:27.529740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.529784 (+ 44us) service_pool.cc:225] Handling call
1026 02:56:31.220825 (+3691041us) ts_tablet_manager.cc:1151] Deleting tablet 741792cf96164695b03de3fb86b12602
1026 02:56:31.220832 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.220918 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3690439}
I20251026 02:56:31.221294 2377 tablet_replica.cc:333] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:31.221531 2377 raft_consensus.cc:2243] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:31.221814 2377 pending_rounds.cc:70] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:31.221894 2377 pending_rounds.cc:77] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888502062661632 op_type: NO_OP noop_request { }
I20251026 02:56:31.222000 2377 raft_consensus.cc:2889] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:31.222102 2377 raft_consensus.cc:2272] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:31.222797 2377 ts_tablet_manager.cc:1916] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:31.223701 2998 raft_consensus.cc:493] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.223852 2998 raft_consensus.cc:515] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.224537 2998 leader_election.cc:290] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:31.225445 2998 raft_consensus.cc:493] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.225587 2998 raft_consensus.cc:515] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.225692 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 8dbc12d49b094dbeadad2e76916ba9af (table default.Table_cdb10d198a394aac9033a98152fd2d94 [id=1d0784496733469fbff42783f90d56d2]) successfully deleted
W20251026 02:56:31.226015 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 2524ee0c964345459c3e62f1e39b8f9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2524ee0c964345459c3e62f1e39b8f9e already in progress: creating tablet
I20251026 02:56:31.226245 2998 leader_election.cc:290] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:31.226262 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ec13c5386abe414ba25589e03d5b850a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec13c5386abe414ba25589e03d5b850a already in progress: creating tablet
W20251026 02:56:31.226490 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 741792cf96164695b03de3fb86b12602 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 741792cf96164695b03de3fb86b12602 already in progress: creating tablet
I20251026 02:56:31.227159 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1370b9ad917344c093e5da3d0e9b1cf0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:31.227418 2118 raft_consensus.cc:2468] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:31.227840 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03da415f3a944a9a9679b3a45295ce35" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:31.227988 2114 raft_consensus.cc:2468] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:31.228884 2195 leader_election.cc:304] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:31.229293 2195 leader_election.cc:304] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:31.213071 2375 consensus_queue.cc:260] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.229519 3028 raft_consensus.cc:2804] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:31.229656 2998 raft_consensus.cc:2804] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:31.229666 3028 raft_consensus.cc:493] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:31.229846 3028 raft_consensus.cc:3060] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.229812 2998 raft_consensus.cc:493] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:31.229944 2998 raft_consensus.cc:3060] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.230258 2375 ts_tablet_manager.cc:1434] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.043s user 0.000s sys 0.003s
I20251026 02:56:31.230756 2376 log.cc:1199] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/9a08e8b7eb1949408f2151f2d7257e9d
I20251026 02:56:31.231236 2376 ts_tablet_manager.cc:1950] T 9a08e8b7eb1949408f2151f2d7257e9d P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:31.231601 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 421) took 1698 ms. Trace:
I20251026 02:56:31.231691 2376 rpcz_store.cc:276] 1026 02:56:29.533607 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.552494 (+ 18887us) service_pool.cc:225] Handling call
1026 02:56:31.021467 (+1468973us) ts_tablet_manager.cc:1151] Deleting tablet 9a08e8b7eb1949408f2151f2d7257e9d
1026 02:56:31.021475 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.114662 (+ 93187us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.213379 (+ 98717us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.231547 (+ 18168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1468760}
I20251026 02:56:31.232267 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 420) took 1727 ms. Trace:
I20251026 02:56:31.232364 2376 rpcz_store.cc:276] 1026 02:56:29.504415 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.504536 (+ 121us) service_pool.cc:225] Handling call
1026 02:56:31.232029 (+1727493us) ts_tablet_manager.cc:1151] Deleting tablet dbf2a1ce78154415b23ab6d0efbc1f52
1026 02:56:31.232040 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.232186 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1663870}
I20251026 02:56:31.232724 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 434) took 1414 ms. Trace:
I20251026 02:56:31.232795 2376 rpcz_store.cc:276] 1026 02:56:29.817958 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.818013 (+ 55us) service_pool.cc:225] Handling call
1026 02:56:31.232571 (+1414558us) ts_tablet_manager.cc:1151] Deleting tablet 0e05eada8742471f87d73f5d960817c0
1026 02:56:31.232579 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.232683 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1414339}
I20251026 02:56:31.233183 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 435) took 1414 ms. Trace:
I20251026 02:56:31.233286 2376 rpcz_store.cc:276] 1026 02:56:29.818476 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.818522 (+ 46us) service_pool.cc:225] Handling call
1026 02:56:31.233037 (+1414515us) ts_tablet_manager.cc:1151] Deleting tablet 79c18ebf56e045b6a52fae8871a9107f
1026 02:56:31.233048 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.233133 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1414264}
I20251026 02:56:31.233642 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 438) took 1402 ms. Trace:
I20251026 02:56:31.233729 2376 rpcz_store.cc:276] 1026 02:56:29.830697 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.830754 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:31.233495 (+1402741us) ts_tablet_manager.cc:1151] Deleting tablet bfa68f2c942c43e2a02e9d333bdfa2a2
1026 02:56:31.233503 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.233580 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1402510}
I20251026 02:56:31.234217 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1370b9ad917344c093e5da3d0e9b1cf0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:31.234405 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03da415f3a944a9a9679b3a45295ce35" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:31.234779 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6e2b159cc3414b619a96fa2c002c764d" } modify_external_catalogs: true
I20251026 02:56:31.234879 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 439) took 1222 ms. Trace:
I20251026 02:56:31.234943 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 9a08e8b7eb1949408f2151f2d7257e9d (table default.Table_5f363b68bc2c4e7685f25e4eefa532a8 [id=43cb7f538f824ae38059645230cc7e94]) successfully deleted
I20251026 02:56:31.234989 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6e2b159cc3414b619a96fa2c002c764d" } modify_external_catalogs: true
I20251026 02:56:31.234961 2376 rpcz_store.cc:276] 1026 02:56:30.012660 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.016501 (+ 3841us) service_pool.cc:225] Handling call
1026 02:56:31.233937 (+1217436us) ts_tablet_manager.cc:1151] Deleting tablet e89a98fcfdbb4011876b0ebc2af91f67
1026 02:56:31.233944 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.234831 (+ 887us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1216292}
W20251026 02:56:31.235261 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet dbf2a1ce78154415b23ab6d0efbc1f52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbf2a1ce78154415b23ab6d0efbc1f52 already in progress: creating tablet
I20251026 02:56:31.235356 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 440) took 1201 ms. Trace:
I20251026 02:56:31.235436 2376 rpcz_store.cc:276] 1026 02:56:30.033414 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.033478 (+ 64us) service_pool.cc:225] Handling call
1026 02:56:31.235220 (+1201742us) ts_tablet_manager.cc:1151] Deleting tablet 95c9fbae62db4850b0f02a2443a3432b
1026 02:56:31.235228 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.235308 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1201491}
W20251026 02:56:31.235522 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0e05eada8742471f87d73f5d960817c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e05eada8742471f87d73f5d960817c0 already in progress: creating tablet
I20251026 02:56:31.235772 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 441) took 1062 ms. Trace:
W20251026 02:56:31.235754 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 79c18ebf56e045b6a52fae8871a9107f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79c18ebf56e045b6a52fae8871a9107f already in progress: creating tablet
I20251026 02:56:31.235860 2376 rpcz_store.cc:276] 1026 02:56:30.172844 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.173107 (+ 263us) service_pool.cc:225] Handling call
1026 02:56:31.235646 (+1062539us) ts_tablet_manager.cc:1151] Deleting tablet c1dd78f088f84a509c761c934c039016
1026 02:56:31.235654 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.235724 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1062211}
I20251026 02:56:31.235780 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c5ded183cc2844e98b226ab83673cea0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:31.236084 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet bfa68f2c942c43e2a02e9d333bdfa2a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfa68f2c942c43e2a02e9d333bdfa2a2 already in progress: creating tablet
I20251026 02:56:31.236121 2376 tablet_replica.cc:333] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:31.236332 2376 raft_consensus.cc:2243] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:31.236523 2376 raft_consensus.cc:2272] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:31.236511 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5ded183cc2844e98b226ab83673cea0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.237316 2376 ts_tablet_manager.cc:1916] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:31.238152 2194 leader_election.cc:343] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:31.238567 2194 leader_election.cc:343] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.238790 3006 raft_consensus.cc:493] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.239037 3006 raft_consensus.cc:515] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:31.239202 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet e89a98fcfdbb4011876b0ebc2af91f67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e89a98fcfdbb4011876b0ebc2af91f67 already in progress: creating tablet
W20251026 02:56:31.239502 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 95c9fbae62db4850b0f02a2443a3432b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95c9fbae62db4850b0f02a2443a3432b already in progress: creating tablet
I20251026 02:56:31.239861 3006 leader_election.cc:290] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:31.239739 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet c1dd78f088f84a509c761c934c039016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1dd78f088f84a509c761c934c039016 already in progress: creating tablet
I20251026 02:56:31.242506 2378 tablet_bootstrap.cc:492] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:31.242712 2378 ts_tablet_manager.cc:1403] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.104s user 0.009s sys 0.054s
I20251026 02:56:31.243695 2378 raft_consensus.cc:359] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.243925 2378 raft_consensus.cc:385] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.244150 2378 raft_consensus.cc:740] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.244498 2378 consensus_queue.cc:260] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.245107 2378 ts_tablet_manager.cc:1434] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:31.247548 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8a66b2dca58445b1877938b41d7b4507" } new_table_name: "default.Table_c30cfa52b6524323bda478613d74178b" modify_external_catalogs: true
I20251026 02:56:31.250650 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e56b259a314422aba35eb129021bbeb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:31.251387 2053 leader_election.cc:343] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.251663 3006 raft_consensus.cc:493] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.251801 3006 raft_consensus.cc:515] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.252439 3006 leader_election.cc:290] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:31.253523 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "122de8616b8f4fcf88971d5738c47e82" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:31.254143 2053 leader_election.cc:343] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.257385 2381 tablet_bootstrap.cc:492] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:31.257611 2381 ts_tablet_manager.cc:1403] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.000s
I20251026 02:56:31.258484 2381 raft_consensus.cc:359] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.258711 2381 raft_consensus.cc:385] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.258811 2381 raft_consensus.cc:740] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.259102 2381 consensus_queue.cc:260] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.259645 2381 ts_tablet_manager.cc:1434] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:31.260052 2381 tablet_bootstrap.cc:492] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:31.262059 2381 tablet_bootstrap.cc:654] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.262562 2378 tablet_bootstrap.cc:492] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:31.263571 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e56b259a314422aba35eb129021bbeb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:31.269008 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e56b259a314422aba35eb129021bbeb) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:31.263928 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "122de8616b8f4fcf88971d5738c47e82" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:31.269562 2053 leader_election.cc:343] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.269716 2053 leader_election.cc:304] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.270126 2934 raft_consensus.cc:2749] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:31.270674 2378 tablet_bootstrap.cc:654] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:31.270758 2053 leader_election.cc:343] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.270870 2053 leader_election.cc:304] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.271242 2934 raft_consensus.cc:2749] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:31.266757 2194 consensus_peers.cc:597] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ec035b1caac450d9ab55e74a061ae38. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:31.272377 2962 consensus_queue.cc:1048] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:31.272936 2962 raft_consensus.cc:493] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.273082 2962 raft_consensus.cc:515] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.273665 2962 leader_election.cc:290] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:31.274225 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e5bb0a6a9f947cf9ce119c1fc91fbf5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:31.274427 1974 raft_consensus.cc:2468] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:31.274865 2194 leader_election.cc:304] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:31.275182 2962 raft_consensus.cc:2804] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:31.275300 2962 raft_consensus.cc:493] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:31.275297 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet ae95654d129441e08fcb3f3b355ffc22 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:31.275393 2962 raft_consensus.cc:3060] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.275915 2375 tablet_bootstrap.cc:492] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:31.276373 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4db275efdd344817b11a2168675cf95e consensus state has no leader
I20251026 02:56:31.278174 2375 tablet_bootstrap.cc:654] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:31.279830 2195 leader_election.cc:343] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9e5bb0a6a9f947cf9ce119c1fc91fbf5
I20251026 02:56:31.298318 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 276bd3b97c97426daf802e1f57feb265
I20251026 02:56:31.299775 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 276bd3b97c97426daf802e1f57feb265 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.299819 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 276bd3b97c97426daf802e1f57feb265 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:31.300704 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7488ff00e4c04293b85e265c4d686fc2" } new_table_name: "default.Table_fc676ef48d7e4c7bbb2fac99cdf17176" modify_external_catalogs: true
I20251026 02:56:31.306107 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 276bd3b97c97426daf802e1f57feb265 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.307852 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e87acf02545f4f63987389f32903aec1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:31.308665 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e87acf02545f4f63987389f32903aec1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:31.318425 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe48472efa444bf580e4074c96a9de92 consensus state has no leader
I20251026 02:56:31.320564 2998 raft_consensus.cc:515] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.321331 2998 leader_election.cc:290] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:31.323441 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03da415f3a944a9a9679b3a45295ce35" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:31.324434 2194 leader_election.cc:343] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.330699 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03da415f3a944a9a9679b3a45295ce35" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:31.330958 2118 raft_consensus.cc:3060] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.333127 3024 tablet.cc:1722] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:31.334233 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2dfe8bcc7f1e4933a4174bdab19cffd5" } modify_external_catalogs: true
I20251026 02:56:31.334491 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2dfe8bcc7f1e4933a4174bdab19cffd5" } modify_external_catalogs: true
I20251026 02:56:31.334875 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c8a0cf8b6d464c81825ff3b2d17afeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:31.335311 3023 tablet.cc:1722] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20251026 02:56:31.335922 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8a0cf8b6d464c81825ff3b2d17afeaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.341622 2375 tablet_bootstrap.cc:492] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:31.341817 2375 ts_tablet_manager.cc:1403] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.003s
I20251026 02:56:31.342221 1661 catalog_manager.cc:5649] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:31.342665 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 358e805370ef410e9d65ee92a8d04f34 (table default.Table_71583d73a49142ae9fec31193fb53542 [id=e86d9fc892a8409590c555e25b27a12f]). Expected version 1 got 0
I20251026 02:56:31.342662 2375 raft_consensus.cc:359] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.342855 2375 raft_consensus.cc:385] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.342917 2375 raft_consensus.cc:740] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.342911 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 865fbd674db34f668fb5a2f5836a9b43 (table default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa]). Expected version 1 got 0
I20251026 02:56:31.343408 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 548a7f9628ac4623b2c2f723f83c1e45 (table default.Table_f8baefeba2b54cb39affb2d4f1b0bef8 [id=d9a6b0b6f3e246a3aa8fbd1167205264]). Expected version 2 got 0
I20251026 02:56:31.346104 2375 consensus_queue.cc:260] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.347055 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 6979455632e54d2081065954771f34d3 (DEFAULT_TABLE table=default.Table_d1fa467b0a5a410aa8905dfabb5b5ea1 [id=a111da525efe4df2a129972120a3871a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.348572 2375 ts_tablet_manager.cc:1434] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251026 02:56:31.348889 2375 tablet_bootstrap.cc:492] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:31.350206 2375 tablet_bootstrap.cc:654] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.357263 2375 tablet_bootstrap.cc:492] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:31.357455 2375 ts_tablet_manager.cc:1403] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20251026 02:56:31.357975 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 6979455632e54d2081065954771f34d3 (DEFAULT_TABLE table=default.Table_d1fa467b0a5a410aa8905dfabb5b5ea1 [id=a111da525efe4df2a129972120a3871a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.358408 2375 raft_consensus.cc:359] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.358613 2375 raft_consensus.cc:385] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.358719 2375 raft_consensus.cc:740] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.358954 2375 consensus_queue.cc:260] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.364437 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 4b5e61afe90e4bc4a2897de21e0b3919
I20251026 02:56:31.364970 1658 catalog_manager.cc:6719] ReplaceTablet: tablet 4b5e61afe90e4bc4a2897de21e0b3919 deleted and replaced by tablet 5b562b7b51f94b268c667650df199fbd
I20251026 02:56:31.365319 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b5e61afe90e4bc4a2897de21e0b3919 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b562b7b51f94b268c667650df199fbd) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:31.365820 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b5e61afe90e4bc4a2897de21e0b3919 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b562b7b51f94b268c667650df199fbd) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.366158 2375 ts_tablet_manager.cc:1434] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251026 02:56:31.366220 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b5e61afe90e4bc4a2897de21e0b3919 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b562b7b51f94b268c667650df199fbd) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.366485 2375 tablet_bootstrap.cc:492] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:31.366798 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet f6d6f024d8b943d5acf69b64bc7c7214 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:31.369609 2375 tablet_bootstrap.cc:654] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.372932 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9a345fa5edbd48d2891b8b40b09cb3f8
I20251026 02:56:31.373994 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a345fa5edbd48d2891b8b40b09cb3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.374433 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a345fa5edbd48d2891b8b40b09cb3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.374461 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a345fa5edbd48d2891b8b40b09cb3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:31.374925 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a3af2695c41a4439bc4092b7cb1b2edf
I20251026 02:56:31.375941 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet a3af2695c41a4439bc4092b7cb1b2edf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:31.376533 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet a3af2695c41a4439bc4092b7cb1b2edf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.376617 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet a3af2695c41a4439bc4092b7cb1b2edf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.381309 2375 tablet_bootstrap.cc:492] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:31.381359 1663 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 049a915b35224cdaa38df77c734a0f01 (table default.Table_1818605e0bc548938337de5d5fc7a781 [id=f1f9d2ad39b34abbbe989bc3fac62b6c]). Expected version 4 got 0
I20251026 02:56:31.381573 2375 ts_tablet_manager.cc:1403] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20251026 02:56:31.381768 1663 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 865fbd674db34f668fb5a2f5836a9b43 (table default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa]). Expected version 1 got 0
I20251026 02:56:31.382120 1663 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 94886ad6d6bb45f39261725169774ac9 (table default.Table_d23ac2c9657b4adeaac9545d5bc9ae41 [id=9e897d33e6374ad4bc3c8acaae512645]). Expected version 1 got 0
I20251026 02:56:31.382350 2375 raft_consensus.cc:359] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.382570 2375 raft_consensus.cc:385] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.382652 2375 raft_consensus.cc:740] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.382404 1663 catalog_manager.cc:5649] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:31.382987 2375 consensus_queue.cc:260] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.383083 1663 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 548a7f9628ac4623b2c2f723f83c1e45 (table default.Table_f8baefeba2b54cb39affb2d4f1b0bef8 [id=d9a6b0b6f3e246a3aa8fbd1167205264]). Expected version 2 got 0
I20251026 02:56:31.386885 2375 ts_tablet_manager.cc:1434] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251026 02:56:31.387260 2375 tablet_bootstrap.cc:492] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:31.387630 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5
I20251026 02:56:31.388799 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:31.391983 2375 tablet_bootstrap.cc:654] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.392099 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.393136 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.401134 1661 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_a6b94cbe15a34811a6a613f0d749a414 [id=c8435cef755d4de69c33f599df1c0f75] alter complete (version 1)
I20251026 02:56:31.403240 3006 raft_consensus.cc:493] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.403474 3006 raft_consensus.cc:515] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.403594 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 115ccccf16b74efc9f24b607cf0c74d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.403911 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet e62064fbdffb4954a3a1227efd6327ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.404130 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 03da415f3a944a9a9679b3a45295ce35 with delete_type TABLET_DATA_DELETED (replaced by tablet a6f9ae327aa4401baf66a737eaa93cf6) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.404155 3006 leader_election.cc:290] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:31.404392 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 86d8e1e21c5f440d8baf76c357d01ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.405040 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "863ce9e1adb444b7b1399b968ea79bb4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:31.405680 2053 leader_election.cc:343] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.405623 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "863ce9e1adb444b7b1399b968ea79bb4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:31.406059 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 8dbc12d49b094dbeadad2e76916ba9af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:31.406605 2053 leader_election.cc:343] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.406746 2053 leader_election.cc:304] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.407044 3006 raft_consensus.cc:2749] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:31.408149 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_908a8748249c46ad964acc6abf54cd58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:31.409116 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_908a8748249c46ad964acc6abf54cd58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.411288 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_02a57a3ced604a78bd1d59268a599261" } new_table_name: "default.Table_fe09be2527a840b09e3e64c27965845d" modify_external_catalogs: true
I20251026 02:56:31.412846 3028 raft_consensus.cc:515] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.413136 2118 raft_consensus.cc:2468] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:31.413589 3028 leader_election.cc:290] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:31.413808 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4f13aec4f4284f51926349af4159512d" } new_table_name: "default.Table_9421cb3b5cd340f5a53162c9855d96b3" modify_external_catalogs: true
I20251026 02:56:31.414112 2376 ts_tablet_manager.cc:1929] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:31.414249 2376 log.cc:1199] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/c1b6257a3c8c4e1a8144ea45ff2a2a96
I20251026 02:56:31.414644 2376 ts_tablet_manager.cc:1950] T c1b6257a3c8c4e1a8144ea45ff2a2a96 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:31.415287 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b389bf6a8e5d4ca39fb2c59f8103652c" } modify_external_catalogs: true
I20251026 02:56:31.415421 2372 ts_tablet_manager.cc:1929] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:31.415445 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b389bf6a8e5d4ca39fb2c59f8103652c" } modify_external_catalogs: true
I20251026 02:56:31.415584 2372 log.cc:1199] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/86d8e1e21c5f440d8baf76c357d01ad2
I20251026 02:56:31.416054 2372 ts_tablet_manager.cc:1950] T 86d8e1e21c5f440d8baf76c357d01ad2 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:31.418303 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6979455632e54d2081065954771f34d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.418550 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_472acbcc3a244422abe7a7477c65c87a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:31.418941 2962 raft_consensus.cc:515] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.419348 2195 leader_election.cc:304] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.419595 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1370b9ad917344c093e5da3d0e9b1cf0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:31.419664 2962 leader_election.cc:290] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:31.419862 2962 raft_consensus.cc:2804] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:31.420192 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6979455632e54d2081065954771f34d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.419800 2118 raft_consensus.cc:3060] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.420528 2377 ts_tablet_manager.cc:1929] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:31.420657 2377 log.cc:1199] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/9156314f98ac4ed9981f6f13fc585043
I20251026 02:56:31.421078 2377 ts_tablet_manager.cc:1950] T 9156314f98ac4ed9981f6f13fc585043 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:31.421950 2962 raft_consensus.cc:697] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:31.422305 2962 consensus_queue.cc:237] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.422981 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1370b9ad917344c093e5da3d0e9b1cf0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:31.423106 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e5bb0a6a9f947cf9ce119c1fc91fbf5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:31.423312 1979 raft_consensus.cc:3060] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.423625 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_38566e9ac15c44b9a28b046e47bf9448"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:31.423846 2961 raft_consensus.cc:493] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.426301 2961 raft_consensus.cc:515] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.427026 2961 leader_election.cc:290] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:31.419113 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_472acbcc3a244422abe7a7477c65c87a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.428285 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e62064fbdffb4954a3a1227efd6327ee" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:31.428459 2195 leader_election.cc:343] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9e5bb0a6a9f947cf9ce119c1fc91fbf5
I20251026 02:56:31.429054 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 2f1872ef9f244a388d25a8085dd53f82 (DEFAULT_TABLE table=default.Table_70e8c0a6af49412e870552049d11eb6c [id=e0a89d776c174aa4928fba81cd3c3974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.429009 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 6979455632e54d2081065954771f34d3 (DEFAULT_TABLE table=default.Table_d1fa467b0a5a410aa8905dfabb5b5ea1 [id=a111da525efe4df2a129972120a3871a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.429437 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f1872ef9f244a388d25a8085dd53f82. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:31.428467 2194 leader_election.cc:343] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:31.430025 2194 leader_election.cc:343] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.431123 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 443) took 1138 ms. Trace:
I20251026 02:56:31.431300 2376 rpcz_store.cc:276] 1026 02:56:30.292866 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.292930 (+ 64us) service_pool.cc:225] Handling call
1026 02:56:31.236051 (+943121us) ts_tablet_manager.cc:1151] Deleting tablet c1b6257a3c8c4e1a8144ea45ff2a2a96
1026 02:56:31.236059 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.324634 (+ 88575us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.413910 (+ 89276us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.431076 (+ 17166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":942844}
I20251026 02:56:31.431555 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 355) took 3887 ms. Trace:
I20251026 02:56:31.431661 2377 rpcz_store.cc:276] 1026 02:56:27.543962 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.544026 (+ 64us) service_pool.cc:225] Handling call
1026 02:56:31.221217 (+3677191us) ts_tablet_manager.cc:1151] Deleting tablet 9156314f98ac4ed9981f6f13fc585043
1026 02:56:31.221226 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.321674 (+100448us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.420370 (+ 98696us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.431521 (+ 11151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3672765}
I20251026 02:56:31.431908 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 442) took 1166 ms. Trace:
I20251026 02:56:31.432055 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 356) took 3879 ms. Trace:
I20251026 02:56:31.432034 2376 rpcz_store.cc:276] 1026 02:56:30.265642 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.303636 (+ 37994us) service_pool.cc:225] Handling call
1026 02:56:31.431702 (+1128066us) ts_tablet_manager.cc:1151] Deleting tablet 99d318aa0d1847c58167bf0af9e75aa2
1026 02:56:31.431715 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.431861 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106729}
I20251026 02:56:31.432153 2377 rpcz_store.cc:276] 1026 02:56:27.552677 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.552728 (+ 51us) service_pool.cc:225] Handling call
1026 02:56:31.431911 (+3879183us) ts_tablet_manager.cc:1151] Deleting tablet a6f9ae327aa4401baf66a737eaa93cf6
1026 02:56:31.431921 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.432022 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3878901}
I20251026 02:56:31.432408 2376 tablet_replica.cc:333] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:31.432443 2377 tablet_replica.cc:333] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:31.432652 2377 raft_consensus.cc:2243] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:31.432685 2376 raft_consensus.cc:2243] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:31.432816 2377 raft_consensus.cc:2272] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:31.432881 2376 raft_consensus.cc:2272] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:31.432883 2195 leader_election.cc:343] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: e62064fbdffb4954a3a1227efd6327ee
I20251026 02:56:31.432979 2195 leader_election.cc:304] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.433422 2377 ts_tablet_manager.cc:1916] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:31.433528 3006 raft_consensus.cc:493] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.433547 2376 ts_tablet_manager.cc:1916] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:31.433702 3006 raft_consensus.cc:515] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.434373 3006 leader_election.cc:290] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:31.434453 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 467) took 1165 ms. Trace:
I20251026 02:56:31.434567 2372 rpcz_store.cc:276] 1026 02:56:30.268761 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.268819 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:31.137467 (+868648us) ts_tablet_manager.cc:1151] Deleting tablet 86d8e1e21c5f440d8baf76c357d01ad2
1026 02:56:31.137477 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.216871 (+ 79394us) spinlock_profiling.cc:243] Waited 60 ms on lock 0x6160001c0354. stack: 00007f8f76bbd9f6 00007f8f76bbd7b0 00007f8f75b993ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8f82a83100 00007f8f83dd0ae3 00007f8f8698d287 00007f8f869c02d4 00007f8f869a0d70 00007f8f869a0afd 0000000000a8867c 00007f8f76c03d7f 00007f8f76c0704b 00007f8f76c06ec8
1026 02:56:31.316733 (+ 99862us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.415265 (+ 98532us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.434405 (+ 19140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138105216,"tablet-delete.queue_time_us":868399}
I20251026 02:56:31.435138 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a08efd37968846d2958e54831488235c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:31.435452 2372 tablet_replica.cc:333] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:31.435703 2372 raft_consensus.cc:2243] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:31.435936 2372 raft_consensus.cc:2272] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:31.436357 2961 raft_consensus.cc:2749] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:31.436550 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a08efd37968846d2958e54831488235c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:31.435736 2053 leader_election.cc:343] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.437582 2372 ts_tablet_manager.cc:1916] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:31.438508 2053 leader_election.cc:343] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.438686 2053 leader_election.cc:304] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.439347 3006 raft_consensus.cc:2749] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:31.433429 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 9156314f98ac4ed9981f6f13fc585043 (table default.Table_4578a4ce81ea42cd878859a8167babff [id=f914c9974ac44d3f84c3d312649fe255]) successfully deleted
I20251026 02:56:31.437029 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 86d8e1e21c5f440d8baf76c357d01ad2 (table default.Table_689f802c9e4b457eb71fb78898fe14b0 [id=05242557de9a4a19948b8591ce73ab28]) successfully deleted
W20251026 02:56:31.448642 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 1eabfc95caa84afc9a8956a7ac13fea6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1eabfc95caa84afc9a8956a7ac13fea6 already in progress: creating tablet
W20251026 02:56:31.448674 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet a6f9ae327aa4401baf66a737eaa93cf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6f9ae327aa4401baf66a737eaa93cf6 already in progress: creating tablet
W20251026 02:56:31.448927 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 4b49c0996c4d41cbaddcdf0b75846bcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b49c0996c4d41cbaddcdf0b75846bcd already in progress: creating tablet
W20251026 02:56:31.449177 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7177931353164b2aa82c00b2807ccd63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7177931353164b2aa82c00b2807ccd63 already in progress: creating tablet
W20251026 02:56:31.449405 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 8fe03778a1e842829cab25c90597a3cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fe03778a1e842829cab25c90597a3cd already in progress: creating tablet
I20251026 02:56:31.450517 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f6a0eda989c8467cbb302a20f18c402e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:31.450631 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet c1b6257a3c8c4e1a8144ea45ff2a2a96 (table default.Table_d23ac2c9657b4adeaac9545d5bc9ae41 [id=9e897d33e6374ad4bc3c8acaae512645]) successfully deleted
W20251026 02:56:31.450891 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 99d318aa0d1847c58167bf0af9e75aa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99d318aa0d1847c58167bf0af9e75aa2 already in progress: creating tablet
W20251026 02:56:31.451143 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6a0eda989c8467cbb302a20f18c402e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.451714 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 586bd6e40a7a4785b71b2a3d542e2411 (DEFAULT_TABLE table=default.Table_b0315e3c5d3f481b9db0d3570ad5a9c2 [id=3a44ddd5789e4958a3e290702eeb4265]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.452322 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 122c81787d5c43d28b1fb639e3542c76 (DEFAULT_TABLE table=default.Table_a4dc7d3d1de04432881624228345e98b [id=f6ff8dff403644e2bfe2800318a3c1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.452548 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 035724949fe142ea93cb998f8fdc4344 (DEFAULT_TABLE table=default.Table_684ec52e10144760980fb35e3d5bad26 [id=55b18e0e84e64241a852055412d02569]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.452757 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 947ad059be4b475f92b5ba208e3dd9e5 (DEFAULT_TABLE table=default.Table_4f13aec4f4284f51926349af4159512d [id=59ce7a857b0d4a78b452b0c05a6c9088]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.453083 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 586bd6e40a7a4785b71b2a3d542e2411 (DEFAULT_TABLE table=default.Table_b0315e3c5d3f481b9db0d3570ad5a9c2 [id=3a44ddd5789e4958a3e290702eeb4265]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.453136 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 035724949fe142ea93cb998f8fdc4344 (DEFAULT_TABLE table=default.Table_684ec52e10144760980fb35e3d5bad26 [id=55b18e0e84e64241a852055412d02569]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.453197 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 7053d454def04264a956aa75ceaad5c9 (DEFAULT_TABLE table=default.Table_3f4de82346384f0faf6482156a0457d1 [id=2a147f9c71b044b9afd63a5865e34c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.453562 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 122c81787d5c43d28b1fb639e3542c76 (DEFAULT_TABLE table=default.Table_a4dc7d3d1de04432881624228345e98b [id=f6ff8dff403644e2bfe2800318a3c1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.453687 2096 tablet_service.cc:1505] Processing CreateTablet for tablet d0f1d6967931482e9de40b0844da6315 (DEFAULT_TABLE table=default.Table_5a3412a706124204a0379462d7628a17 [id=881224ae789e4daab936faafe4765729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.453995 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 947ad059be4b475f92b5ba208e3dd9e5 (DEFAULT_TABLE table=default.Table_4f13aec4f4284f51926349af4159512d [id=59ce7a857b0d4a78b452b0c05a6c9088]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.454134 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 5ddc7972f29a429ea51f6c6c1a074baf (DEFAULT_TABLE table=default.Table_999c4a3cf18e418eaf40b7dbee0e5095 [id=62b36e0fbe964bb5a4c9a2af8f2a70ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.454486 1949 tablet_service.cc:1505] Processing CreateTablet for tablet 7053d454def04264a956aa75ceaad5c9 (DEFAULT_TABLE table=default.Table_3f4de82346384f0faf6482156a0457d1 [id=2a147f9c71b044b9afd63a5865e34c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.454555 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 7f04c9dfc648419798bb527de0d24753 (DEFAULT_TABLE table=default.Table_1eaf8c8b19bf4a62bb69ffe0472d7910 [id=c7887cb3f0e643f5be4e054dbc9759f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.454907 1946 tablet_service.cc:1505] Processing CreateTablet for tablet d0f1d6967931482e9de40b0844da6315 (DEFAULT_TABLE table=default.Table_5a3412a706124204a0379462d7628a17 [id=881224ae789e4daab936faafe4765729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.455364 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 2f1872ef9f244a388d25a8085dd53f82 (DEFAULT_TABLE table=default.Table_70e8c0a6af49412e870552049d11eb6c [id=e0a89d776c174aa4928fba81cd3c3974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.455364 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 5ddc7972f29a429ea51f6c6c1a074baf (DEFAULT_TABLE table=default.Table_999c4a3cf18e418eaf40b7dbee0e5095 [id=62b36e0fbe964bb5a4c9a2af8f2a70ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.455826 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 7f04c9dfc648419798bb527de0d24753 (DEFAULT_TABLE table=default.Table_1eaf8c8b19bf4a62bb69ffe0472d7910 [id=c7887cb3f0e643f5be4e054dbc9759f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.456768 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_76c2c5fa88984a46ac25402fa3d04654"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:31.457463 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76c2c5fa88984a46ac25402fa3d04654 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:31.459028 2194 consensus_peers.cc:597] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:31.459828 2226 tablet_service.cc:1505] Processing CreateTablet for tablet 2f1872ef9f244a388d25a8085dd53f82 (DEFAULT_TABLE table=default.Table_70e8c0a6af49412e870552049d11eb6c [id=e0a89d776c174aa4928fba81cd3c3974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.460654 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 035724949fe142ea93cb998f8fdc4344 (DEFAULT_TABLE table=default.Table_684ec52e10144760980fb35e3d5bad26 [id=55b18e0e84e64241a852055412d02569]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.461229 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 586bd6e40a7a4785b71b2a3d542e2411 (DEFAULT_TABLE table=default.Table_b0315e3c5d3f481b9db0d3570ad5a9c2 [id=3a44ddd5789e4958a3e290702eeb4265]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.461663 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 7053d454def04264a956aa75ceaad5c9 (DEFAULT_TABLE table=default.Table_3f4de82346384f0faf6482156a0457d1 [id=2a147f9c71b044b9afd63a5865e34c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.462213 2241 tablet_service.cc:1505] Processing CreateTablet for tablet d0f1d6967931482e9de40b0844da6315 (DEFAULT_TABLE table=default.Table_5a3412a706124204a0379462d7628a17 [id=881224ae789e4daab936faafe4765729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.462319 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 122c81787d5c43d28b1fb639e3542c76 (DEFAULT_TABLE table=default.Table_a4dc7d3d1de04432881624228345e98b [id=f6ff8dff403644e2bfe2800318a3c1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.462719 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 947ad059be4b475f92b5ba208e3dd9e5 (DEFAULT_TABLE table=default.Table_4f13aec4f4284f51926349af4159512d [id=59ce7a857b0d4a78b452b0c05a6c9088]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.462807 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 5ddc7972f29a429ea51f6c6c1a074baf (DEFAULT_TABLE table=default.Table_999c4a3cf18e418eaf40b7dbee0e5095 [id=62b36e0fbe964bb5a4c9a2af8f2a70ba]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:31.427085 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38566e9ac15c44b9a28b046e47bf9448 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.467489 3030 raft_consensus.cc:493] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.467684 3030 raft_consensus.cc:515] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:31.469811 1916 leader_election.cc:343] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 8dbc12d49b094dbeadad2e76916ba9af
I20251026 02:56:31.470870 3030 leader_election.cc:290] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:31.473595 1650 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 155a23a93358487a9564bb5ac175b4eb (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 155a23a93358487a9564bb5ac175b4eb consensus state has no leader
W20251026 02:56:31.480736 1915 leader_election.cc:343] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 8dbc12d49b094dbeadad2e76916ba9af
I20251026 02:56:31.480914 1915 leader_election.cc:304] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:31.481371 3030 raft_consensus.cc:2749] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:31.489312 3072 raft_consensus.cc:493] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 209968db4dee47c49497a66c3ee750b7)
I20251026 02:56:31.489423 2375 tablet_bootstrap.cc:492] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:31.489566 2375 ts_tablet_manager.cc:1403] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.102s user 0.017s sys 0.026s
I20251026 02:56:31.490451 2375 raft_consensus.cc:359] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.490641 2375 raft_consensus.cc:385] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.490724 2375 raft_consensus.cc:740] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.490928 3072 raft_consensus.cc:515] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:31.491888 3006 raft_consensus.cc:493] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.492058 3006 raft_consensus.cc:515] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.492789 3006 leader_election.cc:290] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:31.493652 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352a284902c94f088e2e6ae006162db0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:31.493841 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352a284902c94f088e2e6ae006162db0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:31.494120 1624 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu pid 2485
I20251026 02:56:31.494305 1673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 4 candidate_status { last_received { term: 3 index: 407 } } ignore_live_leader: false dest_uuid: "209968db4dee47c49497a66c3ee750b7" is_pre_election: true
W20251026 02:56:31.494848 2053 leader_election.cc:343] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.496431 2375 consensus_queue.cc:260] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.497288 2375 ts_tablet_manager.cc:1434] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20251026 02:56:31.498114 2053 leader_election.cc:343] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.498270 2053 leader_election.cc:304] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.498680 3006 raft_consensus.cc:2749] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:29.004648 2720 rpcz_store.cc:276] 1026 02:56:27.411146 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.411205 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:27.418379 (+ 7174us) raft_consensus.cc:1471] Updating replica for 361 ops
1026 02:56:27.541771 (+123392us) raft_consensus.cc:1534] Early marking committed up to index 46
1026 02:56:27.544773 (+ 3002us) raft_consensus.cc:1539] Triggering prepare for 361 ops
1026 02:56:27.545124 (+ 351us) write_op.cc:270] Start()
1026 02:56:27.545186 (+ 62us) write_op.cc:276] Timestamp: P: 1761447380612777 usec, L: 0
1026 02:56:27.548564 (+ 3378us) write_op.cc:270] Start()
1026 02:56:27.548619 (+ 55us) write_op.cc:276] Timestamp: P: 1761447380621986 usec, L: 0
1026 02:56:27.553050 (+ 4431us) write_op.cc:270] Start()
1026 02:56:27.553097 (+ 47us) write_op.cc:276] Timestamp: P: 1761447380623054 usec, L: 0
1026 02:56:27.553354 (+ 257us) write_op.cc:270] Start()
1026 02:56:27.553383 (+ 29us) write_op.cc:276] Timestamp: P: 1761447380623892 usec, L: 0
1026 02:56:27.553574 (+ 191us) write_op.cc:270] Start()
1026 02:56:27.553599 (+ 25us) write_op.cc:276] Timestamp: P: 1761447380640362 usec, L: 0
1026 02:56:27.553766 (+ 167us) write_op.cc:270] Start()
1026 02:56:27.553795 (+ 29us) write_op.cc:276] Timestamp: P: 1761447380723351 usec, L: 0
1026 02:56:27.553951 (+ 156us) write_op.cc:270] Start()
1026 02:56:27.554007 (+ 56us) write_op.cc:276] Timestamp: P: 1761447380724145 usec, L: 0
1026 02:56:27.556270 (+ 2263us) write_op.cc:270] Start()
1026 02:56:27.556301 (+ 31us) write_op.cc:276] Timestamp: P: 1761447380735575 usec, L: 0
1026 02:56:27.556603 (+ 302us) write_op.cc:270] Start()
1026 02:56:27.557816 (+ 1213us) write_op.cc:276] Timestamp: P: 1761447380742971 usec, L: 0
1026 02:56:27.558049 (+ 233us) write_op.cc:270] Start()
1026 02:56:27.558076 (+ 27us) write_op.cc:276] Timestamp: P: 1761447380754791 usec, L: 0
1026 02:56:27.558248 (+ 172us) write_op.cc:270] Start()
1026 02:56:27.558273 (+ 25us) write_op.cc:276] Timestamp: P: 1761447380755670 usec, L: 0
1026 02:56:27.558418 (+ 145us) write_op.cc:270] Start()
1026 02:56:27.558443 (+ 25us) write_op.cc:276] Timestamp: P: 1761447380756406 usec, L: 0
1026 02:56:27.558622 (+ 179us) write_op.cc:270] Start()
1026 02:56:27.558648 (+ 26us) write_op.cc:276] Timestamp: P: 1761447380757097 usec, L: 0
1026 02:56:27.558971 (+ 323us) write_op.cc:270] Start()
1026 02:56:27.559009 (+ 38us) write_op.cc:276] Timestamp: P: 1761447380757800 usec, L: 0
1026 02:56:27.559198 (+ 189us) write_op.cc:270] Start()
1026 02:56:27.559468 (+ 270us) write_op.cc:276] Timestamp: P: 1761447380758484 usec, L: 0
1026 02:56:27.559655 (+ 187us) write_op.cc:270] Start()
1026 02:56:27.559691 (+ 36us) write_op.cc:276] Timestamp: P: 1761447380759149 usec, L: 0
1026 02:56:27.559873 (+ 182us) write_op.cc:270] Start()
1026 02:56:27.559896 (+ 23us) write_op.cc:276] Timestamp: P: 1761447380802583 usec, L: 0
1026 02:56:27.560052 (+ 156us) write_op.cc:270] Start()
1026 02:56:27.560079 (+ 27us) write_op.cc:276] Timestamp: P: 1761447380812370 usec, L: 0
1026 02:56:27.561326 (+ 1247us) write_op.cc:270] Start()
1026 02:56:27.561360 (+ 34us) write_op.cc:276] Timestamp: P: 1761447380825376 usec, L: 0
1026 02:56:27.602830 (+ 41470us) write_op.cc:270] Start()
1026 02:56:27.603163 (+ 333us) write_op.cc:276] Timestamp: P: 1761447380835783 usec, L: 0
1026 02:56:27.639480 (+ 36317us) write_op.cc:270] Start()
1026 02:56:27.639527 (+ 47us) write_op.cc:276] Timestamp: P: 1761447380844606 usec, L: 0
1026 02:56:27.639760 (+ 233us) write_op.cc:270] Start()
1026 02:56:27.639789 (+ 29us) write_op.cc:276] Timestamp: P: 1761447380908008 usec, L: 0
1026 02:56:27.639981 (+ 192us) write_op.cc:270] Start()
1026 02:56:27.640003 (+ 22us) write_op.cc:276] Timestamp: P: 1761447380926661 usec, L: 0
1026 02:56:27.640166 (+ 163us) write_op.cc:270] Start()
1026 02:56:27.640186 (+ 20us) write_op.cc:276] Timestamp: P: 1761447380948665 usec, L: 0
1026 02:56:27.640405 (+ 219us) write_op.cc:270] Start()
1026 02:56:27.640428 (+ 23us) write_op.cc:276] Timestamp: P: 1761447380967036 usec, L: 0
1026 02:56:27.640572 (+ 144us) write_op.cc:270] Start()
1026 02:56:27.640589 (+ 17us) write_op.cc:276] Timestamp: P: 1761447380979398 usec, L: 0
1026 02:56:27.640707 (+ 118us) write_op.cc:270] Start()
1026 02:56:27.640723 (+ 16us) write_op.cc:276] Timestamp: P: 1761447381012967 usec, L: 0
1026 02:56:27.640861 (+ 138us) write_op.cc:270] Start()
1026 02:56:27.640880 (+ 19us) write_op.cc:276] Timestamp: P: 1761447381035988 usec, L: 0
1026 02:56:27.641009 (+ 129us) write_op.cc:270] Start()
1026 02:56:27.641031 (+ 22us) write_op.cc:276] Timestamp: P: 1761447381055477 usec, L: 0
1026 02:56:27.641533 (+ 502us) write_op.cc:270] Start()
1026 02:56:27.641573 (+ 40us) write_op.cc:276] Timestamp: P: 1761447381070199 usec, L: 0
1026 02:56:27.641724 (+ 151us) write_op.cc:270] Start()
1026 02:56:27.641749 (+ 25us) write_op.cc:276] Timestamp: P: 1761447381108426 usec, L: 0
1026 02:56:27.641883 (+ 134us) write_op.cc:270] Start()
1026 02:56:27.641905 (+ 22us) write_op.cc:276] Timestamp: P: 1761447381126824 usec, L: 0
1026 02:56:27.642104 (+ 199us) write_op.cc:270] Start()
1026 02:56:27.642123 (+ 19us) write_op.cc:276] Timestamp: P: 1761447381142788 usec, L: 0
1026 02:56:27.642269 (+ 146us) write_op.cc:270] Start()
1026 02:56:27.642287 (+ 18us) write_op.cc:276] Timestamp: P: 1761447381152277 usec, L: 0
1026 02:56:27.642466 (+ 179us) write_op.cc:270] Start()
1026 02:56:27.642492 (+ 26us) write_op.cc:276] Timestamp: P: 1761447381219343 usec, L: 0
1026 02:56:27.642981 (+ 489us) write_op.cc:270] Start()
1026 02:56:27.643010 (+ 29us) write_op.cc:276] Timestamp: P: 1761447381221660 usec, L: 0
1026 02:56:27.643157 (+ 147us) write_op.cc:270] Start()
1026 02:56:27.643191 (+ 34us) write_op.cc:276] Timestamp: P: 1761447381222515 usec, L: 0
1026 02:56:27.643373 (+ 182us) write_op.cc:270] Start()
1026 02:56:27.643397 (+ 24us) write_op.cc:276] Timestamp: P: 1761447381235808 usec, L: 0
1026 02:56:27.650670 (+ 7273us) write_op.cc:270] Start()
1026 02:56:27.650708 (+ 38us) write_op.cc:276] Timestamp: P: 1761447381236843 usec, L: 0
1026 02:56:27.650883 (+ 175us) write_op.cc:270] Start()
1026 02:56:27.650904 (+ 21us) write_op.cc:276] Timestamp: P: 1761447381237382 usec, L: 0
1026 02:56:27.655729 (+ 4825us) write_op.cc:270] Start()
1026 02:56:27.655766 (+ 37us) write_op.cc:276] Timestamp: P: 1761447381237985 usec, L: 0
1026 02:56:27.655948 (+ 182us) write_op.cc:270] Start()
1026 02:56:27.655969 (+ 21us) write_op.cc:276] Timestamp: P: 1761447381238511 usec, L: 0
1026 02:56:27.656129 (+ 160us) write_op.cc:270] Start()
1026 02:56:27.656152 (+ 23us) write_op.cc:276] Timestamp: P: 1761447381239007 usec, L: 0
1026 02:56:27.656327 (+ 175us) write_op.cc:270] Start()
1026 02:56:27.656351 (+ 24us) write_op.cc:276] Timestamp: P: 1761447381259283 usec, L: 0
1026 02:56:27.656487 (+ 136us) write_op.cc:270] Start()
1026 02:56:27.661645 (+ 5158us) write_op.cc:276] Timestamp: P: 1761447381260492 usec, L: 0
1026 02:56:27.661922 (+ 277us) write_op.cc:270] Start()
1026 02:56:27.661959 (+ 37us) write_op.cc:276] Timestamp: P: 1761447381261026 usec, L: 0
1026 02:56:27.662091 (+ 132us) write_op.cc:270] Start()
1026 02:56:27.662110 (+ 19us) write_op.cc:276] Timestamp: P: 1761447381301692 usec, L: 0
1026 02:56:27.662233 (+ 123us) write_op.cc:270] Start()
1026 02:56:27.662260 (+ 27us) write_op.cc:276] Timestamp: P: 1761447381304801 usec, L: 0
1026 02:56:27.662400 (+ 140us) write_op.cc:270] Start()
1026 02:56:27.662418 (+ 18us) write_op.cc:276] Timestamp: P: 1761447381317984 usec, L: 0
1026 02:56:27.662594 (+ 176us) write_op.cc:270] Start()
1026 02:56:27.662611 (+ 17us) write_op.cc:276] Timestamp: P: 1761447381333388 usec, L: 0
1026 02:56:27.662717 (+ 106us) write_op.cc:270] Start()
1026 02:56:27.662733 (+ 16us) write_op.cc:276] Timestamp: P: 1761447381333996 usec, L: 0
1026 02:56:27.662835 (+ 102us) write_op.cc:270] Start()
1026 02:56:27.662851 (+ 16us) write_op.cc:276] Timestamp: P: 1761447381336874 usec, L: 0
1026 02:56:27.662963 (+ 112us) write_op.cc:270] Start()
1026 02:56:27.662979 (+ 16us) write_op.cc:276] Timestamp: P: 1761447381462276 usec, L: 0
1026 02:56:27.663089 (+ 110us) write_op.cc:270] Start()
1026 02:56:27.663108 (+ 19us) write_op.cc:276] Timestamp: P: 1761447381499456 usec, L: 0
1026 02:56:27.663232 (+ 124us) write_op.cc:270] Start()
1026 02:56:27.663261 (+ 29us) write_op.cc:276] Timestamp: P: 1761447381501443 usec, L: 0
1026 02:56:27.663395 (+ 134us) write_op.cc:270] Start()
1026 02:56:27.663415 (+ 20us) write_op.cc:276] Timestamp: P: 1761447381510578 usec, L: 0
1026 02:56:27.663540 (+ 125us) write_op.cc:270] Start()
1026 02:56:27.663559 (+ 19us) write_op.cc:276] Timestamp: P: 1761447381511835 usec, L: 0
1026 02:56:27.663694 (+ 135us) write_op.cc:270] Start()
1026 02:56:27.663732 (+ 38us) write_op.cc:276] Timestamp: P: 1761447381512691 usec, L: 0
1026 02:56:27.663871 (+ 139us) write_op.cc:270] Start()
1026 02:56:27.663890 (+ 19us) write_op.cc:276] Timestamp: P: 1761447381599983 usec, L: 0
1026 02:56:27.664030 (+ 140us) write_op.cc:270] Start()
1026 02:56:27.664073 (+ 43us) write_op.cc:276] Timestamp: P: 1761447381602747 usec, L: 0
1026 02:56:27.664280 (+ 207us) write_op.cc:270] Start()
1026 02:56:27.664305 (+ 25us) write_op.cc:276] Timestamp: P: 1761447381696466 usec, L: 0
1026 02:56:27.780439 (+116134us) write_op.cc:270] Start()
1026 02:56:27.780491 (+ 52us) write_op.cc:276] Timestamp: P: 1761447381730130 usec, L: 0
1026 02:56:27.780766 (+ 275us) write_op.cc:270] Start()
1026 02:56:27.780795 (+ 29us) write_op.cc:276] Timestamp: P: 1761447381796295 usec, L: 0
1026 02:56:27.780967 (+ 172us) write_op.cc:270] Start()
1026 02:56:27.780992 (+ 25us) write_op.cc:276] Timestamp: P: 1761447381827494 usec, L: 0
1026 02:56:27.781164 (+ 172us) write_op.cc:270] Start()
1026 02:56:27.781200 (+ 36us) write_op.cc:276] Timestamp: P: 1761447381830678 usec, L: 0
1026 02:56:27.781362 (+ 162us) write_op.cc:270] Start()
1026 02:56:27.781405 (+ 43us) write_op.cc:276] Timestamp: P: 1761447381831986 usec, L: 0
1026 02:56:27.781559 (+ 154us) write_op.cc:270] Start()
1026 02:56:27.781582 (+ 23us) write_op.cc:276] Timestamp: P: 1761447381857167 usec, L: 0
1026 02:56:27.781710 (+ 128us) write_op.cc:270] Start()
1026 02:56:27.781727 (+ 17us) write_op.cc:276] Timestamp: P: 1761447381858652 usec, L: 0
1026 02:56:27.781828 (+ 101us) write_op.cc:270] Start()
1026 02:56:27.781843 (+ 15us) write_op.cc:276] Timestamp: P: 1761447381860703 usec, L: 0
1026 02:56:27.781957 (+ 114us) write_op.cc:270] Start()
1026 02:56:27.781980 (+ 23us) write_op.cc:276] Timestamp: P: 1761447381861783 usec, L: 0
1026 02:56:27.782097 (+ 117us) write_op.cc:270] Start()
1026 02:56:27.782114 (+ 17us) write_op.cc:276] Timestamp: P: 1761447381880987 usec, L: 0
1026 02:56:27.782246 (+ 132us) write_op.cc:270] Start()
1026 02:56:27.782264 (+ 18us) write_op.cc:276] Timestamp: P: 1761447381882829 usec, L: 0
1026 02:56:27.782537 (+ 273us) write_op.cc:270] Start()
1026 02:56:27.782560 (+ 23us) write_op.cc:276] Timestamp: P: 1761447381899066 usec, L: 0
1026 02:56:27.782694 (+ 134us) write_op.cc:270] Start()
1026 02:56:27.782713 (+ 19us) write_op.cc:276] Timestamp: P: 1761447381904731 usec, L: 0
1026 02:56:27.782854 (+ 141us) write_op.cc:270] Start()
1026 02:56:27.782874 (+ 20us) write_op.cc:276] Timestamp: P: 1761447381906806 usec, L: 0
1026 02:56:27.783006 (+ 132us) write_op.cc:270] Start()
1026 02:56:27.783192 (+ 186us) write_op.cc:276] Timestamp: P: 1761447381913994 usec, L: 0
1026 02:56:27.783346 (+ 154us) write_op.cc:270] Start()
1026 02:56:27.783368 (+ 22us) write_op.cc:276] Timestamp: P: 1761447381924546 usec, L: 0
1026 02:56:27.783510 (+ 142us) write_op.cc:270] Start()
1026 02:56:27.783529 (+ 19us) write_op.cc:276] Timestamp: P: 1761447381948206 usec, L: 0
1026 02:56:27.783650 (+ 121us) write_op.cc:270] Start()
1026 02:56:27.783714 (+ 64us) write_op.cc:276] Timestamp: P: 1761447381958326 usec, L: 0
1026 02:56:27.783853 (+ 139us) write_op.cc:270] Start()
1026 02:56:27.783869 (+ 16us) write_op.cc:276] Timestamp: P: 1761447381959374 usec, L: 0
1026 02:56:27.783981 (+ 112us) write_op.cc:270] Start()
1026 02:56:27.783999 (+ 18us) write_op.cc:276] Timestamp: P: 1761447381960386 usec, L: 0
1026 02:56:27.784103 (+ 104us) write_op.cc:270] Start()
1026 02:56:27.784120 (+ 17us) write_op.cc:276] Timestamp: P: 1761447381961495 usec, L: 0
1026 02:56:27.784255 (+ 135us) write_op.cc:270] Start()
1026 02:56:27.784272 (+ 17us) write_op.cc:276] Timestamp: P: 1761447381983292 usec, L: 0
1026 02:56:27.784412 (+ 140us) write_op.cc:270] Start()
1026 02:56:27.784433 (+ 21us) write_op.cc:276] Timestamp: P: 1761447381996961 usec, L: 0
1026 02:56:27.784548 (+ 115us) write_op.cc:270] Start()
1026 02:56:27.784565 (+ 17us) write_op.cc:276] Timestamp: P: 1761447381998080 usec, L: 0
1026 02:56:27.784685 (+ 120us) write_op.cc:270] Start()
1026 02:56:27.784702 (+ 17us) write_op.cc:276] Timestamp: P: 1761447381998933 usec, L: 0
1026 02:56:27.784828 (+ 126us) write_op.cc:270] Start()
1026 02:56:27.784854 (+ 26us) write_op.cc:276] Timestamp: P: 1761447382049199 usec, L: 0
1026 02:56:27.785007 (+ 153us) write_op.cc:270] Start()
1026 02:56:27.785036 (+ 29us) write_op.cc:276] Timestamp: P: 1761447382061642 usec, L: 0
1026 02:56:27.785227 (+ 191us) write_op.cc:270] Start()
1026 02:56:27.785252 (+ 25us) write_op.cc:276] Timestamp: P: 1761447382095058 usec, L: 0
1026 02:56:27.785389 (+ 137us) write_op.cc:270] Start()
1026 02:56:27.785410 (+ 21us) write_op.cc:276] Timestamp: P: 1761447382103017 usec, L: 0
1026 02:56:27.785540 (+ 130us) write_op.cc:270] Start()
1026 02:56:27.785560 (+ 20us) write_op.cc:276] Timestamp: P: 1761447382169826 usec, L: 0
1026 02:56:27.785698 (+ 138us) write_op.cc:270] Start()
1026 02:56:27.785720 (+ 22us) write_op.cc:276] Timestamp: P: 1761447382179343 usec, L: 0
1026 02:56:27.785873 (+ 153us) write_op.cc:270] Start()
1026 02:56:27.785897 (+ 24us) write_op.cc:276] Timestamp: P: 1761447382188128 usec, L: 0
1026 02:56:27.786043 (+ 146us) write_op.cc:270] Start()
1026 02:56:27.786063 (+ 20us) write_op.cc:276] Timestamp: P: 1761447382204311 usec, L: 0
1026 02:56:27.786167 (+ 104us) write_op.cc:270] Start()
1026 02:56:27.786195 (+ 28us) write_op.cc:276] Timestamp: P: 1761447382219603 usec, L: 0
1026 02:56:27.786370 (+ 175us) write_op.cc:270] Start()
1026 02:56:27.786393 (+ 23us) write_op.cc:276] Timestamp: P: 1761447382237277 usec, L: 0
1026 02:56:27.786539 (+ 146us) write_op.cc:270] Start()
1026 02:56:27.786562 (+ 23us) write_op.cc:276] Timestamp: P: 1761447382247657 usec, L: 0
1026 02:56:27.786700 (+ 138us) write_op.cc:270] Start()
1026 02:56:27.786723 (+ 23us) write_op.cc:276] Timestamp: P: 1761447382274446 usec, L: 0
1026 02:56:27.786870 (+ 147us) write_op.cc:270] Start()
1026 02:56:27.786889 (+ 19us) write_op.cc:276] Timestamp: P: 1761447382279142 usec, L: 0
1026 02:56:27.787062 (+ 173us) write_op.cc:270] Start()
1026 02:56:27.787087 (+ 25us) write_op.cc:276] Timestamp: P: 1761447382303638 usec, L: 0
1026 02:56:27.787218 (+ 131us) write_op.cc:270] Start()
1026 02:56:27.787239 (+ 21us) write_op.cc:276] Timestamp: P: 1761447382313696 usec, L: 0
1026 02:56:27.787373 (+ 134us) write_op.cc:270] Start()
1026 02:56:27.787390 (+ 17us) write_op.cc:276] Timestamp: P: 1761447382331486 usec, L: 0
1026 02:56:27.787496 (+ 106us) write_op.cc:270] Start()
1026 02:56:27.787514 (+ 18us) write_op.cc:276] Timestamp: P: 1761447382342906 usec, L: 0
1026 02:56:27.941136 (+153622us) write_op.cc:270] Start()
1026 02:56:27.941194 (+ 58us) write_op.cc:276] Timestamp: P: 1761447382362197 usec, L: 0
1026 02:56:27.941435 (+ 241us) write_op.cc:270] Start()
1026 02:56:27.941457 (+ 22us) write_op.cc:276] Timestamp: P: 1761447382372976 usec, L: 0
1026 02:56:27.941578 (+ 121us) write_op.cc:270] Start()
1026 02:56:27.941597 (+ 19us) write_op.cc:276] Timestamp: P: 1761447382390821 usec, L: 0
1026 02:56:27.952579 (+ 10982us) write_op.cc:270] Start()
1026 02:56:27.952631 (+ 52us) write_op.cc:276] Timestamp: P: 1761447382410424 usec, L: 0
1026 02:56:27.952837 (+ 206us) write_op.cc:270] Start()
1026 02:56:27.952863 (+ 26us) write_op.cc:276] Timestamp: P: 1761447382428141 usec, L: 0
1026 02:56:27.953030 (+ 167us) write_op.cc:270] Start()
1026 02:56:27.953055 (+ 25us) write_op.cc:276] Timestamp: P: 1761447382444073 usec, L: 0
1026 02:56:27.953219 (+ 164us) write_op.cc:270] Start()
1026 02:56:27.953241 (+ 22us) write_op.cc:276] Timestamp: P: 1761447382462097 usec, L: 0
1026 02:56:27.953383 (+ 142us) write_op.cc:270] Start()
1026 02:56:27.953408 (+ 25us) write_op.cc:276] Timestamp: P: 1761447382485986 usec, L: 0
1026 02:56:27.953584 (+ 176us) write_op.cc:270] Start()
1026 02:56:27.953608 (+ 24us) write_op.cc:276] Timestamp: P: 1761447382497438 usec, L: 0
1026 02:56:27.953767 (+ 159us) write_op.cc:270] Start()
1026 02:56:27.953790 (+ 23us) write_op.cc:276] Timestamp: P: 1761447382513775 usec, L: 0
1026 02:56:27.953920 (+ 130us) write_op.cc:270] Start()
1026 02:56:27.953936 (+ 16us) write_op.cc:276] Timestamp: P: 1761447382533389 usec, L: 0
1026 02:56:27.954237 (+ 301us) write_op.cc:270] Start()
1026 02:56:27.954264 (+ 27us) write_op.cc:276] Timestamp: P: 1761447382549922 usec, L: 0
1026 02:56:27.954455 (+ 191us) write_op.cc:270] Start()
1026 02:56:27.954480 (+ 25us) write_op.cc:276] Timestamp: P: 1761447382565507 usec, L: 0
1026 02:56:27.954624 (+ 144us) write_op.cc:270] Start()
1026 02:56:27.954648 (+ 24us) write_op.cc:276] Timestamp: P: 1761447382580423 usec, L: 0
1026 02:56:27.954798 (+ 150us) write_op.cc:270] Start()
1026 02:56:27.954821 (+ 23us) write_op.cc:276] Timestamp: P: 1761447382592884 usec, L: 0
1026 02:56:27.954961 (+ 140us) write_op.cc:270] Start()
1026 02:56:27.954984 (+ 23us) write_op.cc:276] Timestamp: P: 1761447382607312 usec, L: 0
1026 02:56:27.955153 (+ 169us) write_op.cc:270] Start()
1026 02:56:27.955178 (+ 25us) write_op.cc:276] Timestamp: P: 1761447382617804 usec, L: 0
1026 02:56:27.955350 (+ 172us) write_op.cc:270] Start()
1026 02:56:27.955373 (+ 23us) write_op.cc:276] Timestamp: P: 1761447382636381 usec, L: 0
1026 02:56:27.955520 (+ 147us) write_op.cc:270] Start()
1026 02:56:27.955544 (+ 24us) write_op.cc:276] Timestamp: P: 1761447382650765 usec, L: 0
1026 02:56:27.955692 (+ 148us) write_op.cc:270] Start()
1026 02:56:27.955716 (+ 24us) write_op.cc:276] Timestamp: P: 1761447382668674 usec, L: 0
1026 02:56:27.955863 (+ 147us) write_op.cc:270] Start()
1026 02:56:27.955885 (+ 22us) write_op.cc:276] Timestamp: P: 1761447382678148 usec, L: 0
1026 02:56:27.956050 (+ 165us) write_op.cc:270] Start()
1026 02:56:27.956086 (+ 36us) write_op.cc:276] Timestamp: P: 1761447382687919 usec, L: 0
1026 02:56:28.055048 (+ 98962us) write_op.cc:270] Start()
1026 02:56:28.055114 (+ 66us) write_op.cc:276] Timestamp: P: 1761447382695727 usec, L: 0
1026 02:56:28.055422 (+ 308us) write_op.cc:270] Start()
1026 02:56:28.055452 (+ 30us) write_op.cc:276] Timestamp: P: 1761447382709884 usec, L: 0
1026 02:56:28.055654 (+ 202us) write_op.cc:270] Start()
1026 02:56:28.055680 (+ 26us) write_op.cc:276] Timestamp: P: 1761447382723968 usec, L: 0
1026 02:56:28.055871 (+ 191us) write_op.cc:270] Start()
1026 02:56:28.055895 (+ 24us) write_op.cc:276] Timestamp: P: 1761447382763981 usec, L: 0
1026 02:56:28.056096 (+ 201us) write_op.cc:270] Start()
1026 02:56:28.056123 (+ 27us) write_op.cc:276] Timestamp: P: 1761447382780497 usec, L: 0
1026 02:56:28.063765 (+ 7642us) write_op.cc:270] Start()
1026 02:56:28.063877 (+ 112us) write_op.cc:276] Timestamp: P: 1761447382805392 usec, L: 0
1026 02:56:28.064158 (+ 281us) write_op.cc:270] Start()
1026 02:56:28.064180 (+ 22us) write_op.cc:276] Timestamp: P: 1761447382825872 usec, L: 0
1026 02:56:28.064391 (+ 211us) write_op.cc:270] Start()
1026 02:56:28.064418 (+ 27us) write_op.cc:276] Timestamp: P: 1761447382826537 usec, L: 0
1026 02:56:28.064581 (+ 163us) write_op.cc:270] Start()
1026 02:56:28.064606 (+ 25us) write_op.cc:276] Timestamp: P: 1761447382827125 usec, L: 0
1026 02:56:28.064752 (+ 146us) write_op.cc:270] Start()
1026 02:56:28.064776 (+ 24us) write_op.cc:276] Timestamp: P: 1761447382827703 usec, L: 0
1026 02:56:28.064936 (+ 160us) write_op.cc:270] Start()
1026 02:56:28.064959 (+ 23us) write_op.cc:276] Timestamp: P: 1761447382828502 usec, L: 0
1026 02:56:28.065611 (+ 652us) write_op.cc:270] Start()
1026 02:56:28.065647 (+ 36us) write_op.cc:276] Timestamp: P: 1761447382829040 usec, L: 0
1026 02:56:28.065842 (+ 195us) write_op.cc:270] Start()
1026 02:56:28.065867 (+ 25us) write_op.cc:276] Timestamp: P: 1761447382969612 usec, L: 0
1026 02:56:28.066039 (+ 172us) write_op.cc:270] Start()
1026 02:56:28.066064 (+ 25us) write_op.cc:276] Timestamp: P: 1761447382970291 usec, L: 0
1026 02:56:28.111997 (+ 45933us) write_op.cc:270] Start()
1026 02:56:28.112048 (+ 51us) write_op.cc:276] Timestamp: P: 1761447382971679 usec, L: 0
1026 02:56:28.112352 (+ 304us) write_op.cc:270] Start()
1026 02:56:28.112388 (+ 36us) write_op.cc:276] Timestamp: P: 1761447382984048 usec, L: 0
1026 02:56:28.112546 (+ 158us) write_op.cc:270] Start()
1026 02:56:28.112565 (+ 19us) write_op.cc:276] Timestamp: P: 1761447382984786 usec, L: 0
1026 02:56:28.112676 (+ 111us) write_op.cc:270] Start()
1026 02:56:28.112695 (+ 19us) write_op.cc:276] Timestamp: P: 1761447382985368 usec, L: 0
1026 02:56:28.120640 (+ 7945us) write_op.cc:270] Start()
1026 02:56:28.120675 (+ 35us) write_op.cc:276] Timestamp: P: 1761447382985905 usec, L: 0
1026 02:56:28.120832 (+ 157us) write_op.cc:270] Start()
1026 02:56:28.120853 (+ 21us) write_op.cc:276] Timestamp: P: 1761447382987817 usec, L: 0
1026 02:56:28.121002 (+ 149us) write_op.cc:270] Start()
1026 02:56:28.121024 (+ 22us) write_op.cc:276] Timestamp: P: 1761447383021448 usec, L: 0
1026 02:56:28.121170 (+ 146us) write_op.cc:270] Start()
1026 02:56:28.121193 (+ 23us) write_op.cc:276] Timestamp: P: 1761447383069624 usec, L: 0
1026 02:56:28.121353 (+ 160us) write_op.cc:270] Start()
1026 02:56:28.121374 (+ 21us) write_op.cc:276] Timestamp: P: 1761447383100934 usec, L: 0
1026 02:56:28.121495 (+ 121us) write_op.cc:270] Start()
1026 02:56:28.121534 (+ 39us) write_op.cc:276] Timestamp: P: 1761447383115422 usec, L: 0
1026 02:56:28.121712 (+ 178us) write_op.cc:270] Start()
1026 02:56:28.121732 (+ 20us) write_op.cc:276] Timestamp: P: 1761447383116436 usec, L: 0
1026 02:56:28.121912 (+ 180us) write_op.cc:270] Start()
1026 02:56:28.121937 (+ 25us) write_op.cc:276] Timestamp: P: 1761447383126304 usec, L: 0
1026 02:56:28.122081 (+ 144us) write_op.cc:270] Start()
1026 02:56:28.122104 (+ 23us) write_op.cc:276] Timestamp: P: 1761447383145499 usec, L: 0
1026 02:56:28.122258 (+ 154us) write_op.cc:270] Start()
1026 02:56:28.122279 (+ 21us) write_op.cc:276] Timestamp: P: 1761447383154337 usec, L: 0
1026 02:56:28.122410 (+ 131us) write_op.cc:270] Start()
1026 02:56:28.122427 (+ 17us) write_op.cc:276] Timestamp: P: 1761447383168459 usec, L: 0
1026 02:56:28.122557 (+ 130us) write_op.cc:270] Start()
1026 02:56:28.122586 (+ 29us) write_op.cc:276] Timestamp: P: 1761447383188575 usec, L: 0
1026 02:56:28.122722 (+ 136us) write_op.cc:270] Start()
1026 02:56:28.122743 (+ 21us) write_op.cc:276] Timestamp: P: 1761447383211016 usec, L: 0
1026 02:56:28.122882 (+ 139us) write_op.cc:270] Start()
1026 02:56:28.122902 (+ 20us) write_op.cc:276] Timestamp: P: 1761447383227791 usec, L: 0
1026 02:56:28.123089 (+ 187us) write_op.cc:270] Start()
1026 02:56:28.123113 (+ 24us) write_op.cc:276] Timestamp: P: 1761447383240764 usec, L: 0
1026 02:56:28.123296 (+ 183us) write_op.cc:270] Start()
1026 02:56:28.123322 (+ 26us) write_op.cc:276] Timestamp: P: 1761447383299090 usec, L: 0
1026 02:56:28.137745 (+ 14423us) write_op.cc:270] Start()
1026 02:56:28.137782 (+ 37us) write_op.cc:276] Timestamp: P: 1761447383329844 usec, L: 0
1026 02:56:28.138004 (+ 222us) write_op.cc:270] Start()
1026 02:56:28.138026 (+ 22us) write_op.cc:276] Timestamp: P: 1761447383348568 usec, L: 0
1026 02:56:28.138151 (+ 125us) write_op.cc:270] Start()
1026 02:56:28.138172 (+ 21us) write_op.cc:276] Timestamp: P: 1761447383382690 usec, L: 0
1026 02:56:28.138295 (+ 123us) write_op.cc:270] Start()
1026 02:56:28.138313 (+ 18us) write_op.cc:276] Timestamp: P: 1761447383488430 usec, L: 0
1026 02:56:28.138469 (+ 156us) write_op.cc:270] Start()
1026 02:56:28.138495 (+ 26us) write_op.cc:276] Timestamp: P: 1761447383509257 usec, L: 0
1026 02:56:28.139852 (+ 1357us) write_op.cc:270] Start()
1026 02:56:28.139884 (+ 32us) write_op.cc:276] Timestamp: P: 1761447383525726 usec, L: 0
1026 02:56:28.140054 (+ 170us) write_op.cc:270] Start()
1026 02:56:28.140075 (+ 21us) write_op.cc:276] Timestamp: P: 1761447383541435 usec, L: 0
1026 02:56:28.140282 (+ 207us) write_op.cc:270] Start()
1026 02:56:28.140379 (+ 97us) write_op.cc:276] Timestamp: P: 1761447383554543 usec, L: 0
1026 02:56:28.140564 (+ 185us) write_op.cc:270] Start()
1026 02:56:28.140583 (+ 19us) write_op.cc:276] Timestamp: P: 1761447383562565 usec, L: 0
1026 02:56:28.140754 (+ 171us) write_op.cc:270] Start()
1026 02:56:28.140783 (+ 29us) write_op.cc:276] Timestamp: P: 1761447383563299 usec, L: 0
1026 02:56:28.140974 (+ 191us) write_op.cc:270] Start()
1026 02:56:28.148131 (+ 7157us) write_op.cc:276] Timestamp: P: 1761447383563835 usec, L: 0
1026 02:56:28.148411 (+ 280us) write_op.cc:270] Start()
1026 02:56:28.148440 (+ 29us) write_op.cc:276] Timestamp: P: 1761447383564441 usec, L: 0
1026 02:56:28.148586 (+ 146us) write_op.cc:270] Start()
1026 02:56:28.148607 (+ 21us) write_op.cc:276] Timestamp: P: 1761447383564969 usec, L: 0
1026 02:56:28.148747 (+ 140us) write_op.cc:270] Start()
1026 02:56:28.148770 (+ 23us) write_op.cc:276] Timestamp: P: 1761447383565503 usec, L: 0
1026 02:56:28.148937 (+ 167us) write_op.cc:270] Start()
1026 02:56:28.148965 (+ 28us) write_op.cc:276] Timestamp: P: 1761447383566036 usec, L: 0
1026 02:56:28.149140 (+ 175us) write_op.cc:270] Start()
1026 02:56:28.149158 (+ 18us) write_op.cc:276] Timestamp: P: 1761447383566585 usec, L: 0
1026 02:56:28.149293 (+ 135us) write_op.cc:270] Start()
1026 02:56:28.149314 (+ 21us) write_op.cc:276] Timestamp: P: 1761447383567126 usec, L: 0
1026 02:56:28.149429 (+ 115us) write_op.cc:270] Start()
1026 02:56:28.149446 (+ 17us) write_op.cc:276] Timestamp: P: 1761447383601739 usec, L: 0
1026 02:56:28.149580 (+ 134us) write_op.cc:270] Start()
1026 02:56:28.149600 (+ 20us) write_op.cc:276] Timestamp: P: 1761447383625118 usec, L: 0
1026 02:56:28.156979 (+ 7379us) write_op.cc:270] Start()
1026 02:56:28.157026 (+ 47us) write_op.cc:276] Timestamp: P: 1761447383637383 usec, L: 0
1026 02:56:28.157271 (+ 245us) write_op.cc:270] Start()
1026 02:56:28.157293 (+ 22us) write_op.cc:276] Timestamp: P: 1761447383640546 usec, L: 0
1026 02:56:28.157441 (+ 148us) write_op.cc:270] Start()
1026 02:56:28.157459 (+ 18us) write_op.cc:276] Timestamp: P: 1761447383641167 usec, L: 0
1026 02:56:28.157580 (+ 121us) write_op.cc:270] Start()
1026 02:56:28.157597 (+ 17us) write_op.cc:276] Timestamp: P: 1761447383641762 usec, L: 0
1026 02:56:28.157728 (+ 131us) write_op.cc:270] Start()
1026 02:56:28.157756 (+ 28us) write_op.cc:276] Timestamp: P: 1761447383642293 usec, L: 0
1026 02:56:28.157936 (+ 180us) write_op.cc:270] Start()
1026 02:56:28.157961 (+ 25us) write_op.cc:276] Timestamp: P: 1761447383647186 usec, L: 0
1026 02:56:28.158074 (+ 113us) write_op.cc:270] Start()
1026 02:56:28.158091 (+ 17us) write_op.cc:276] Timestamp: P: 1761447383647819 usec, L: 0
1026 02:56:28.158207 (+ 116us) write_op.cc:270] Start()
1026 02:56:28.158242 (+ 35us) write_op.cc:276] Timestamp: P: 1761447383716442 usec, L: 0
1026 02:56:28.158381 (+ 139us) write_op.cc:270] Start()
1026 02:56:28.158399 (+ 18us) write_op.cc:276] Timestamp: P: 1761447383789773 usec, L: 0
1026 02:56:28.158518 (+ 119us) write_op.cc:270] Start()
1026 02:56:28.158537 (+ 19us) write_op.cc:276] Timestamp: P: 1761447383796863 usec, L: 0
1026 02:56:28.158702 (+ 165us) write_op.cc:270] Start()
1026 02:56:28.158721 (+ 19us) write_op.cc:276] Timestamp: P: 1761447383853706 usec, L: 0
1026 02:56:28.158849 (+ 128us) write_op.cc:270] Start()
1026 02:56:28.158867 (+ 18us) write_op.cc:276] Timestamp: P: 1761447383862606 usec, L: 0
1026 02:56:28.159029 (+ 162us) write_op.cc:270] Start()
1026 02:56:28.159054 (+ 25us) write_op.cc:276] Timestamp: P: 1761447383863376 usec, L: 0
1026 02:56:28.159184 (+ 130us) write_op.cc:270] Start()
1026 02:56:28.159202 (+ 18us) write_op.cc:276] Timestamp: P: 1761447383863979 usec, L: 0
1026 02:56:28.159345 (+ 143us) write_op.cc:270] Start()
1026 02:56:28.159363 (+ 18us) write_op.cc:276] Timestamp: P: 1761447383864533 usec, L: 0
1026 02:56:28.159477 (+ 114us) write_op.cc:270] Start()
1026 02:56:28.159493 (+ 16us) write_op.cc:276] Timestamp: P: 1761447383867300 usec, L: 0
1026 02:56:28.159623 (+ 130us) write_op.cc:270] Start()
1026 02:56:28.159646 (+ 23us) write_op.cc:276] Timestamp: P: 1761447383963643 usec, L: 0
1026 02:56:28.159765 (+ 119us) write_op.cc:270] Start()
1026 02:56:28.159787 (+ 22us) write_op.cc:276] Timestamp: P: 1761447383964346 usec, L: 0
1026 02:56:28.159928 (+ 141us) write_op.cc:270] Start()
1026 02:56:28.159946 (+ 18us) write_op.cc:276] Timestamp: P: 1761447
I20251026 02:56:31.510346 3030 raft_consensus.cc:493] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.510557 3030 raft_consensus.cc:515] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.511204 3030 leader_election.cc:290] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:31.512724 1916 leader_election.cc:343] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 7845ebd3e9ca4897b8fb19a48a010315
W20251026 02:56:31.513007 1915 leader_election.cc:343] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 7845ebd3e9ca4897b8fb19a48a010315
I20251026 02:56:31.513120 1915 leader_election.cc:304] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:31.513513 3030 raft_consensus.cc:2749] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:31.514626 1979 raft_consensus.cc:2468] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:31.514634 2118 raft_consensus.cc:2468] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:31.515141 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f1872ef9f244a388d25a8085dd53f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.515833 2195 leader_election.cc:304] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.516306 2962 raft_consensus.cc:2804] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:31.516396 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6979455632e54d2081065954771f34d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.517096 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 035724949fe142ea93cb998f8fdc4344. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.517565 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 586bd6e40a7a4785b71b2a3d542e2411. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.518240 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122c81787d5c43d28b1fb639e3542c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.518664 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 035724949fe142ea93cb998f8fdc4344. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.519125 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 586bd6e40a7a4785b71b2a3d542e2411. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.519538 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ddc7972f29a429ea51f6c6c1a074baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.519915 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f1d6967931482e9de40b0844da6315. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.520321 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 947ad059be4b475f92b5ba208e3dd9e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.520694 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7053d454def04264a956aa75ceaad5c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.520941 1949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7053d454def04264a956aa75ceaad5c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.521042 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122c81787d5c43d28b1fb639e3542c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.524353 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ddc7972f29a429ea51f6c6c1a074baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.524677 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7053d454def04264a956aa75ceaad5c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.525084 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f1872ef9f244a388d25a8085dd53f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.525560 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 947ad059be4b475f92b5ba208e3dd9e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.525604 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 035724949fe142ea93cb998f8fdc4344. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.526034 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f04c9dfc648419798bb527de0d24753. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.526468 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ddc7972f29a429ea51f6c6c1a074baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.526886 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f1d6967931482e9de40b0844da6315. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.527313 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 947ad059be4b475f92b5ba208e3dd9e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.527766 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122c81787d5c43d28b1fb639e3542c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.528204 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 586bd6e40a7a4785b71b2a3d542e2411. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.528504 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f04c9dfc648419798bb527de0d24753. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.532392 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f1d6967931482e9de40b0844da6315. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.527778 3006 raft_consensus.cc:493] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.537700 2194 leader_election.cc:304] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:31.537693 3006 raft_consensus.cc:515] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:31.537971 2194 raft_consensus.cc:2686] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251026 02:56:31.538384 3006 leader_election.cc:290] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:31.543905 2962 raft_consensus.cc:697] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:31.544745 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63a63b796191465aab9a63d3ffb0554d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:31.545815 3030 raft_consensus.cc:493] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.545949 3030 raft_consensus.cc:515] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.546550 3030 leader_election.cc:290] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:31.547174 3072 leader_election.cc:290] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 209968db4dee47c49497a66c3ee750b7 (127.1.150.62:44525), 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655)
W20251026 02:56:31.548638 2053 leader_election.cc:343] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.565671 2962 consensus_queue.cc:237] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:31.566219 1916 leader_election.cc:343] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 155a23a93358487a9564bb5ac175b4eb
I20251026 02:56:31.570062 1742 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 4 candidate_status { last_received { term: 3 index: 407 } } ignore_live_leader: false dest_uuid: "647c0de4ce394caca75179fe11658fc8" is_pre_election: true
I20251026 02:56:31.570227 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63a63b796191465aab9a63d3ffb0554d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:31.570940 2644 leader_election.cc:304] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 751b719b75ce4bc2a88a19091230ecb7; no voters: 209968db4dee47c49497a66c3ee750b7, 647c0de4ce394caca75179fe11658fc8
I20251026 02:56:31.571514 3072 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251026 02:56:31.572067 2053 leader_election.cc:343] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:31.572163 1915 leader_election.cc:343] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 155a23a93358487a9564bb5ac175b4eb
I20251026 02:56:31.572257 2053 leader_election.cc:304] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.572371 1915 leader_election.cc:304] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:31.572556 3006 raft_consensus.cc:2749] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:31.573524 1650 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.61:37655, user_credentials={real_user=slave}} blocked reactor thread for 62305.3us
I20251026 02:56:31.580313 3030 raft_consensus.cc:2749] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:31.590838 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:31.591118 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: This master's current role is: LEADER
I20251026 02:56:31.594022 2381 tablet_bootstrap.cc:492] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:31.594213 2381 ts_tablet_manager.cc:1403] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.334s user 0.025s sys 0.023s
I20251026 02:56:31.595031 2381 raft_consensus.cc:359] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.595199 2381 raft_consensus.cc:385] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.595268 2381 raft_consensus.cc:740] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.595513 2381 consensus_queue.cc:260] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.595983 2381 ts_tablet_manager.cc:1434] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:31.596307 2381 tablet_bootstrap.cc:492] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:31.598003 2381 tablet_bootstrap.cc:654] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.602869 2375 tablet_bootstrap.cc:492] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:31.604936 2375 tablet_bootstrap.cc:654] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.613241 2376 ts_tablet_manager.cc:1929] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251026 02:56:31.625910 1663 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 94886ad6d6bb45f39261725169774ac9 (table default.Table_d23ac2c9657b4adeaac9545d5bc9ae41 [id=9e897d33e6374ad4bc3c8acaae512645], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 94886ad6d6bb45f39261725169774ac9 consensus state has no leader
I20251026 02:56:31.626588 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet c31d2704d1f14ddead69c794cc631ec1 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:31.627794 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 049a915b35224cdaa38df77c734a0f01 (table default.Table_1818605e0bc548938337de5d5fc7a781 [id=f1f9d2ad39b34abbbe989bc3fac62b6c]). Expected version 4 got 0
I20251026 02:56:31.631973 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5a3412a706124204a0379462d7628a17" } new_table_name: "default.Table_53a2c0226802442a8954f3903443b574" modify_external_catalogs: true
I20251026 02:56:31.632779 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ac31a485be77410ca1fafabe643f37a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:31.633437 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac31a485be77410ca1fafabe643f37a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.641963 2381 tablet_bootstrap.cc:492] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:31.642197 2381 ts_tablet_manager.cc:1403] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20251026 02:56:31.643087 2381 raft_consensus.cc:359] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.643330 2381 raft_consensus.cc:385] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.643419 2381 raft_consensus.cc:740] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.643668 2381 consensus_queue.cc:260] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.647991 2381 ts_tablet_manager.cc:1434] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251026 02:56:31.648409 2381 tablet_bootstrap.cc:492] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:31.649767 2376 log.cc:1199] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/5580ac28a98c4ec4a1d83fb9f9bf25ee
I20251026 02:56:31.650437 2376 ts_tablet_manager.cc:1950] T 5580ac28a98c4ec4a1d83fb9f9bf25ee P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:31.650684 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet f85c2745a371455a8dda67f31a19c9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.651176 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 444) took 1297 ms. Trace:
I20251026 02:56:31.651284 2376 rpcz_store.cc:276] 1026 02:56:30.352871 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.359655 (+ 6784us) service_pool.cc:225] Handling call
1026 02:56:31.432337 (+1072682us) ts_tablet_manager.cc:1151] Deleting tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee
1026 02:56:31.432349 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.516864 (+ 84515us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.613103 (+ 96239us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.650781 (+ 37678us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1072400}
I20251026 02:56:31.651948 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 5580ac28a98c4ec4a1d83fb9f9bf25ee (table default.Table_d2dcec579c684b92a92d98fb59ee3da1 [id=4f7edfb6d75c4a0b94ca99bb9cfe14f3]) successfully deleted
I20251026 02:56:31.652473 2376 tablet_replica.cc:333] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:31.652690 2376 raft_consensus.cc:2243] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:31.653120 2376 raft_consensus.cc:2272] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:31.653638 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 115ccccf16b74efc9f24b607cf0c74d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.653718 2376 ts_tablet_manager.cc:1916] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:31.655239 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e56b259a314422aba35eb129021bbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.658474 2381 tablet_bootstrap.cc:654] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.669420 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 0efdcc36ff5b47379e0a76512e05d379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.673260 2381 tablet_bootstrap.cc:492] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:31.673482 2381 ts_tablet_manager.cc:1403] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.003s
I20251026 02:56:31.674345 2381 raft_consensus.cc:359] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.674728 2381 raft_consensus.cc:385] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.674823 2381 raft_consensus.cc:740] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.675127 2381 consensus_queue.cc:260] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.675429 3028 raft_consensus.cc:493] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.675581 3028 raft_consensus.cc:515] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.676304 3028 leader_election.cc:290] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:31.677279 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d90e62ae86b149b291dc29cf7442c6bc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:31.677462 2381 ts_tablet_manager.cc:1434] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20251026 02:56:31.677912 2194 leader_election.cc:343] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.678088 1979 raft_consensus.cc:3060] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.678315 2381 tablet_bootstrap.cc:492] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:31.681286 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d90e62ae86b149b291dc29cf7442c6bc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:31.681540 2118 raft_consensus.cc:2468] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:31.686645 2195 leader_election.cc:304] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.688364 2381 tablet_bootstrap.cc:654] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.689463 2962 raft_consensus.cc:2804] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:31.691522 3006 consensus_queue.cc:1048] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:31.691695 2802 generic_service.cc:196] Checking for leaks (request via RPC)
I20251026 02:56:31.693081 2378 tablet_bootstrap.cc:492] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:31.693331 2378 ts_tablet_manager.cc:1403] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.431s user 0.029s sys 0.038s
I20251026 02:56:31.695091 2378 raft_consensus.cc:359] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.695369 2378 raft_consensus.cc:385] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.695606 2378 raft_consensus.cc:740] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.695909 2378 consensus_queue.cc:260] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.696640 2375 tablet_bootstrap.cc:492] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:31.696856 2375 ts_tablet_manager.cc:1403] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.094s user 0.004s sys 0.003s
I20251026 02:56:31.697163 2378 ts_tablet_manager.cc:1434] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:31.697736 2375 raft_consensus.cc:359] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.697934 2378 tablet_bootstrap.cc:492] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:31.697955 2375 raft_consensus.cc:385] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.698037 2375 raft_consensus.cc:740] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.698319 2375 consensus_queue.cc:260] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.698861 2375 ts_tablet_manager.cc:1434] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:31.700989 2378 tablet_bootstrap.cc:654] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.702173 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 7f04c9dfc648419798bb527de0d24753 (DEFAULT_TABLE table=default.Table_1eaf8c8b19bf4a62bb69ffe0472d7910 [id=c7887cb3f0e643f5be4e054dbc9759f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:31.703730 2375 tablet_bootstrap.cc:492] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:31.706033 2375 tablet_bootstrap.cc:654] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.708292 2372 ts_tablet_manager.cc:1929] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:31.708452 2372 log.cc:1199] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/98fe44aab33b40eba819386c3bffca46
I20251026 02:56:31.708837 2372 ts_tablet_manager.cc:1950] T 98fe44aab33b40eba819386c3bffca46 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:31.714408 2377 ts_tablet_manager.cc:1929] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:31.714574 2377 log.cc:1199] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/9e5bb0a6a9f947cf9ce119c1fc91fbf5
I20251026 02:56:31.714926 2377 ts_tablet_manager.cc:1950] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:31.724331 3030 raft_consensus.cc:493] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.724318 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 98fe44aab33b40eba819386c3bffca46 (table default.Table_481575c8f05747c2954c9c79a0ebde9c [id=91d789aff2464f73b60e03b52126a396]) successfully deleted
I20251026 02:56:31.724535 3030 raft_consensus.cc:515] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.725474 3030 leader_election.cc:290] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:31.726104 1916 leader_election.cc:343] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: c570e5f9a93c45bda32ec92c68794714
I20251026 02:56:31.726526 2372 tablet_replica.cc:333] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:31.726536 1915 leader_election.cc:343] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: c570e5f9a93c45bda32ec92c68794714
I20251026 02:56:31.726661 1915 leader_election.cc:304] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:31.726727 2372 raft_consensus.cc:2243] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:31.726925 2372 raft_consensus.cc:2272] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:31.727034 3030 raft_consensus.cc:2749] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:31.729489 2381 tablet_bootstrap.cc:492] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:31.729667 2381 ts_tablet_manager.cc:1403] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.009s
I20251026 02:56:31.730516 2381 raft_consensus.cc:359] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.730718 2381 raft_consensus.cc:385] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.730814 2381 raft_consensus.cc:740] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.731105 2381 consensus_queue.cc:260] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.731613 2381 ts_tablet_manager.cc:1434] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:31.731920 2381 tablet_bootstrap.cc:492] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:31.732196 2372 ts_tablet_manager.cc:1916] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:31.737265 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f04c9dfc648419798bb527de0d24753. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:31.737743 3023 tablet.cc:1722] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:31.741037 2962 raft_consensus.cc:493] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:31.741230 2962 raft_consensus.cc:3060] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.742833 3024 tablet.cc:1722] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251026 02:56:31.751065 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9cd10092aac40068a1c7f85e514fd37 consensus state has no leader
W20251026 02:56:31.753409 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 947ad059be4b475f92b5ba208e3dd9e5 consensus state has no leader
I20251026 02:56:31.762449 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c985fc44cad2462f8e82eadf887551c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:31.763165 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c985fc44cad2462f8e82eadf887551c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.764703 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_766ba994f1b5469cb050efb60fef1c42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:31.765259 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_766ba994f1b5469cb050efb60fef1c42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.766592 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7ba0c9a1ed6e49f09387b3e13faee903" } new_table_name: "default.Table_0335b328855349de9d1c12fa3beedd0a" modify_external_catalogs: true
I20251026 02:56:31.768038 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet 209fa71af13142a58efb9f3dab3eecd3 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:31.768728 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_70dab64d9fe341f987afbff383e463d9" } modify_external_catalogs: true
I20251026 02:56:31.768909 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_70dab64d9fe341f987afbff383e463d9" } modify_external_catalogs: true
I20251026 02:56:31.769524 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet c31d2704d1f14ddead69c794cc631ec1
I20251026 02:56:31.770128 1663 catalog_manager.cc:6719] ReplaceTablet: tablet c31d2704d1f14ddead69c794cc631ec1 deleted and replaced by tablet de9124fe1c8945d9b9565cb732bef20e
I20251026 02:56:31.770432 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_43e64ec30d194fa89fa789b94321a269" } new_table_name: "default.Table_47994972d3e0469c9acc96120aad4e1f" modify_external_catalogs: true
I20251026 02:56:31.770558 2381 tablet_bootstrap.cc:654] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:31.771876 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0f1d6967931482e9de40b0844da6315 consensus state has no leader
I20251026 02:56:31.772094 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 21b936fa4f6a4ff6af321971a30cccbe from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:31.776500 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7713a2385fb24bb7bfecfaa78176c128" } modify_external_catalogs: true
I20251026 02:56:31.776710 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7713a2385fb24bb7bfecfaa78176c128" } modify_external_catalogs: true
I20251026 02:56:31.780733 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet c31d2704d1f14ddead69c794cc631ec1 with delete_type TABLET_DATA_DELETED (replaced by tablet de9124fe1c8945d9b9565cb732bef20e) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:31.782490 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet c31d2704d1f14ddead69c794cc631ec1 with delete_type TABLET_DATA_DELETED (replaced by tablet de9124fe1c8945d9b9565cb732bef20e) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.783218 3065 consensus_queue.cc:1048] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:31.784921 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 357) took 4231 ms. Trace:
I20251026 02:56:31.785046 2377 rpcz_store.cc:276] 1026 02:56:27.553205 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.553254 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:31.432393 (+3879139us) ts_tablet_manager.cc:1151] Deleting tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5
1026 02:56:31.432401 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.612666 (+180265us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.712420 (+ 99754us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.784849 (+ 72429us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23,"tablet-delete.queue_time_us":3878745}
I20251026 02:56:31.785646 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 358) took 4231 ms. Trace:
I20251026 02:56:31.785751 2377 rpcz_store.cc:276] 1026 02:56:27.553788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.553833 (+ 45us) service_pool.cc:225] Handling call
1026 02:56:31.785436 (+4231603us) ts_tablet_manager.cc:1151] Deleting tablet a012d868a78f4324a92db83c81931b9d
1026 02:56:31.785452 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.785587 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4231167}
I20251026 02:56:31.786185 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 359) took 4231 ms. Trace:
I20251026 02:56:31.786274 2377 rpcz_store.cc:276] 1026 02:56:27.554400 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.554437 (+ 37us) service_pool.cc:225] Handling call
1026 02:56:31.786024 (+4231587us) ts_tablet_manager.cc:1151] Deleting tablet 8c1e63ac8c5645f3ba1578792c0a0dae
1026 02:56:31.786035 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.786130 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4230881}
I20251026 02:56:31.786706 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 361) took 4169 ms. Trace:
I20251026 02:56:31.786830 2377 rpcz_store.cc:276] 1026 02:56:27.617358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.620050 (+ 2692us) service_pool.cc:225] Handling call
1026 02:56:31.786543 (+4166493us) ts_tablet_manager.cc:1151] Deleting tablet cfc3aed64ad0439cb22fd6815c973d87
1026 02:56:31.786554 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.786651 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4166212}
I20251026 02:56:31.787232 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 362) took 4169 ms. Trace:
I20251026 02:56:31.787323 2377 rpcz_store.cc:276] 1026 02:56:27.617464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.620368 (+ 2904us) service_pool.cc:225] Handling call
1026 02:56:31.787074 (+4166706us) ts_tablet_manager.cc:1151] Deleting tablet d43b6f22dd4f4f9c9657f21514550312
1026 02:56:31.787086 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.787175 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4166550}
I20251026 02:56:31.787701 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 365) took 4104 ms. Trace:
W20251026 02:56:31.787698 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 351cd60b076a49949f03640c981845c4 consensus state has no leader
I20251026 02:56:31.787799 2377 rpcz_store.cc:276] 1026 02:56:27.683038 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.694742 (+ 11704us) service_pool.cc:225] Handling call
1026 02:56:31.787555 (+4092813us) ts_tablet_manager.cc:1151] Deleting tablet 3c3050e72d42417ab58a05f7e3aa56b3
1026 02:56:31.787565 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.787653 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4092497}
I20251026 02:56:31.788156 2377 tablet_replica.cc:333] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:31.788695 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3b00d3dd0fe54a4b867f4dcf830d98a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:31.788806 2934 raft_consensus.cc:493] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.788969 2934 raft_consensus.cc:515] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:31.789238 2053 consensus_peers.cc:597] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:31.789331 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b00d3dd0fe54a4b867f4dcf830d98a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.789657 2934 leader_election.cc:290] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:31.790935 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94886ad6d6bb45f39261725169774ac9" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:31.793259 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8dcca91177f5452e816e2080bceadb93" } modify_external_catalogs: true
I20251026 02:56:31.793498 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8dcca91177f5452e816e2080bceadb93" } modify_external_catalogs: true
I20251026 02:56:31.795883 3030 raft_consensus.cc:493] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.796077 3030 raft_consensus.cc:515] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.796808 3030 leader_election.cc:290] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:31.797542 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "903e65fa8c214dc9a50cb97cab101d22" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:31.799106 2053 leader_election.cc:343] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:31.799769 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85f9fcae83a04c6e907c7e9c8cd95dbe consensus state has no leader
I20251026 02:56:31.800371 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_de1a357fbcb3425da0335f0ef9772de9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:31.802485 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "903e65fa8c214dc9a50cb97cab101d22" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:31.805102 1915 leader_election.cc:304] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:31.805203 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94886ad6d6bb45f39261725169774ac9" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:31.805413 2259 raft_consensus.cc:2468] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:31.805631 3030 raft_consensus.cc:2749] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:31.806535 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 (table default.Table_25607539f64440cb8ff2c24a0a3a3bdd [id=5dbfb5d912ba433baa46bd52bbf6dbee]) successfully deleted
W20251026 02:56:31.806913 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet a012d868a78f4324a92db83c81931b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a012d868a78f4324a92db83c81931b9d already in progress: creating tablet
W20251026 02:56:31.807179 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 8c1e63ac8c5645f3ba1578792c0a0dae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c1e63ac8c5645f3ba1578792c0a0dae already in progress: creating tablet
I20251026 02:56:31.807303 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 4011930a277d410f863a1558416c4b0f
W20251026 02:56:31.807420 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet cfc3aed64ad0439cb22fd6815c973d87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfc3aed64ad0439cb22fd6815c973d87 already in progress: creating tablet
W20251026 02:56:31.807637 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet d43b6f22dd4f4f9c9657f21514550312 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43b6f22dd4f4f9c9657f21514550312 already in progress: creating tablet
I20251026 02:56:31.807958 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3f562a6edad942ad94cb473dddea9cd5" } modify_external_catalogs: true
I20251026 02:56:31.808154 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3f562a6edad942ad94cb473dddea9cd5" } modify_external_catalogs: true
I20251026 02:56:31.808751 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet c31d2704d1f14ddead69c794cc631ec1 with delete_type TABLET_DATA_DELETED (replaced by tablet de9124fe1c8945d9b9565cb732bef20e) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.809312 2381 tablet_bootstrap.cc:492] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:31.809471 2381 ts_tablet_manager.cc:1403] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.002s
I20251026 02:56:31.810388 2381 raft_consensus.cc:359] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.810581 2381 raft_consensus.cc:385] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.810676 2381 raft_consensus.cc:740] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.810909 2381 consensus_queue.cc:260] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.810986 2962 raft_consensus.cc:515] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.811483 2381 ts_tablet_manager.cc:1434] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:31.811623 2962 leader_election.cc:290] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:31.811841 2381 tablet_bootstrap.cc:492] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:31.814618 2381 tablet_bootstrap.cc:654] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.815834 3030 raft_consensus.cc:493] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.816020 3030 raft_consensus.cc:515] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.816754 3030 leader_election.cc:290] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:31.817473 2376 ts_tablet_manager.cc:1929] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:31.817680 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d90e62ae86b149b291dc29cf7442c6bc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:31.818291 2194 leader_election.cc:343] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.818281 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4011930a277d410f863a1558416c4b0f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:31.817770 2376 log.cc:1199] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/ac19c5a847454a23ab16831198ff2a20
I20251026 02:56:31.819279 2376 ts_tablet_manager.cc:1950] T ac19c5a847454a23ab16831198ff2a20 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:31.822454 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d90e62ae86b149b291dc29cf7442c6bc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:31.822717 2118 raft_consensus.cc:3060] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.827639 2381 tablet_bootstrap.cc:492] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:31.827913 2381 ts_tablet_manager.cc:1403] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
W20251026 02:56:31.829562 2194 consensus_peers.cc:597] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:31.829644 2053 leader_election.cc:304] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.830317 2934 raft_consensus.cc:2804] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:31.830231 2381 raft_consensus.cc:359] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.830449 2934 raft_consensus.cc:493] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:31.830576 2934 raft_consensus.cc:3060] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.830597 2381 raft_consensus.cc:385] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.830729 2381 raft_consensus.cc:740] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.831175 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4011930a277d410f863a1558416c4b0f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:31.831252 2381 consensus_queue.cc:260] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.832361 2381 ts_tablet_manager.cc:1434] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:31.832829 2381 tablet_bootstrap.cc:492] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:31.833277 1915 leader_election.cc:304] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:31.833745 3030 raft_consensus.cc:2749] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:31.834466 2377 raft_consensus.cc:2243] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:31.834870 2377 pending_rounds.cc:70] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:31.835011 2377 pending_rounds.cc:77] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888501657415680 op_type: NO_OP noop_request { }
I20251026 02:56:31.835251 2377 raft_consensus.cc:2889] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:31.835436 2377 raft_consensus.cc:2272] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:31.836105 2377 ts_tablet_manager.cc:1916] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:31.844659 2381 tablet_bootstrap.cc:654] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.848603 3065 raft_consensus.cc:493] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.848790 3065 raft_consensus.cc:515] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.849438 3065 leader_election.cc:290] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:31.850090 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f85c2745a371455a8dda67f31a19c9d5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:31.850207 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5b95c2f1040441baa94ad346e22b4c72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:31.850852 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f85c2745a371455a8dda67f31a19c9d5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:31.851133 2114 raft_consensus.cc:2468] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:31.851696 2195 leader_election.cc:304] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
W20251026 02:56:31.851946 2194 leader_election.cc:343] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.852630 3065 raft_consensus.cc:2804] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:31.852726 3065 raft_consensus.cc:493] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:31.852844 3065 raft_consensus.cc:3060] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:31.860991 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b95c2f1040441baa94ad346e22b4c72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:31.861227 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de1a357fbcb3425da0335f0ef9772de9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:31.870051 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 4011930a277d410f863a1558416c4b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:31.872529 2381 tablet_bootstrap.cc:492] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:31.872707 2381 ts_tablet_manager.cc:1403] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.005s
I20251026 02:56:31.873443 2381 raft_consensus.cc:359] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.873629 2381 raft_consensus.cc:385] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.873708 2381 raft_consensus.cc:740] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.873935 2381 consensus_queue.cc:260] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.874429 2381 ts_tablet_manager.cc:1434] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:31.874689 2381 tablet_bootstrap.cc:492] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:31.875325 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 4011930a277d410f863a1558416c4b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:31.876698 3007 raft_consensus.cc:493] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.876889 3007 raft_consensus.cc:515] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.877611 3007 leader_election.cc:290] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:31.878223 2381 tablet_bootstrap.cc:654] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.878221 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fe0e06a86e241bc91a1db4219fe25bc" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:31.878518 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fe0e06a86e241bc91a1db4219fe25bc" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:31.879281 2053 leader_election.cc:343] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:31.879892 2053 leader_election.cc:343] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.880102 2053 leader_election.cc:304] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.880575 3007 raft_consensus.cc:2749] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:31.896595 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 4011930a277d410f863a1558416c4b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:31.899428 1974 raft_consensus.cc:3060] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.909027 3096 tablet.cc:1722] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:31.913365 2372 ts_tablet_manager.cc:1929] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:31.913547 2372 log.cc:1199] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/122de8616b8f4fcf88971d5738c47e82
I20251026 02:56:31.913970 2372 ts_tablet_manager.cc:1950] T 122de8616b8f4fcf88971d5738c47e82 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:31.916456 3065 raft_consensus.cc:515] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.919768 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f85c2745a371455a8dda67f31a19c9d5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:31.920027 2114 raft_consensus.cc:3060] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.922329 3065 leader_election.cc:290] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:31.922516 3028 raft_consensus.cc:493] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:31.922787 3028 raft_consensus.cc:515] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.923421 3028 leader_election.cc:290] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:31.925738 2118 raft_consensus.cc:2468] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:31.926450 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0efdcc36ff5b47379e0a76512e05d379" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:31.927209 2195 leader_election.cc:304] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:31.927575 3028 raft_consensus.cc:2804] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:31.927682 3028 raft_consensus.cc:697] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:31.927947 3028 consensus_queue.cc:237] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.926601 2375 tablet_bootstrap.cc:492] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:31.926718 2119 raft_consensus.cc:2468] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:31.930063 2375 ts_tablet_manager.cc:1403] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.226s user 0.021s sys 0.048s
I20251026 02:56:31.930666 2195 leader_election.cc:304] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:31.930881 2375 raft_consensus.cc:359] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.931061 2375 raft_consensus.cc:385] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.931157 2375 raft_consensus.cc:740] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.931183 3065 raft_consensus.cc:2804] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:31.931346 3065 raft_consensus.cc:493] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:31.931453 3065 raft_consensus.cc:3060] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:31.931433 2375 consensus_queue.cc:260] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.934571 2375 ts_tablet_manager.cc:1434] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:31.934893 2375 tablet_bootstrap.cc:492] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:31.936625 2375 tablet_bootstrap.cc:654] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.940053 2378 tablet_bootstrap.cc:492] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:31.940289 2378 ts_tablet_manager.cc:1403] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.242s user 0.006s sys 0.035s
I20251026 02:56:31.941072 2378 raft_consensus.cc:359] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.945669 2378 raft_consensus.cc:385] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.945770 2378 raft_consensus.cc:740] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.946017 2378 consensus_queue.cc:260] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.946434 2378 ts_tablet_manager.cc:1434] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251026 02:56:31.948026 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 79c18ebf56e045b6a52fae8871a9107f from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:31.955431 2375 tablet_bootstrap.cc:492] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:31.955634 2375 ts_tablet_manager.cc:1403] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20251026 02:56:31.956655 2375 raft_consensus.cc:359] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.956889 2375 raft_consensus.cc:385] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.956974 2375 raft_consensus.cc:740] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.957227 2375 consensus_queue.cc:260] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:31.957890 2375 ts_tablet_manager.cc:1434] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:31.958285 2375 tablet_bootstrap.cc:492] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:31.959952 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 5caeb11322fe48348ebb523967152187 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:31.960235 2375 tablet_bootstrap.cc:654] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.965595 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 445) took 1509 ms. Trace:
I20251026 02:56:31.965816 2376 rpcz_store.cc:276] 1026 02:56:30.403789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.404005 (+ 216us) service_pool.cc:225] Handling call
1026 02:56:31.651573 (+1247568us) ts_tablet_manager.cc:1151] Deleting tablet ac19c5a847454a23ab16831198ff2a20
1026 02:56:31.651582 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.727310 (+ 75728us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.817267 (+ 89957us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:31.912892 (+ 95625us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1246945}
I20251026 02:56:31.967216 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet ac19c5a847454a23ab16831198ff2a20 (table default.Table_f8a501269b994d8c8864349d12974e16 [id=a48fa20d35e44980b60317a9c93b42d5]) successfully deleted
I20251026 02:56:31.972229 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0efdcc36ff5b47379e0a76512e05d379" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:31.973003 2194 leader_election.cc:343] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.973429 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f85c2745a371455a8dda67f31a19c9d5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:31.974403 2194 leader_election.cc:343] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:31.973860 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 460) took 1111 ms. Trace:
I20251026 02:56:31.975351 2376 rpcz_store.cc:276] 1026 02:56:30.862657 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.862715 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:31.966102 (+1103387us) ts_tablet_manager.cc:1151] Deleting tablet 1eabfc95caa84afc9a8956a7ac13fea6
1026 02:56:31.973653 (+ 7551us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.973801 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1103170}
I20251026 02:56:31.975855 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 461) took 1073 ms. Trace:
I20251026 02:56:31.975975 2376 rpcz_store.cc:276] 1026 02:56:30.901949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.902029 (+ 80us) service_pool.cc:225] Handling call
1026 02:56:31.975669 (+1073640us) ts_tablet_manager.cc:1151] Deleting tablet 4b49c0996c4d41cbaddcdf0b75846bcd
1026 02:56:31.975683 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.975820 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1073367}
I20251026 02:56:31.976380 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 462) took 1069 ms. Trace:
I20251026 02:56:31.976480 2376 rpcz_store.cc:276] 1026 02:56:30.907068 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.907126 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:31.976204 (+1069078us) ts_tablet_manager.cc:1151] Deleting tablet 7177931353164b2aa82c00b2807ccd63
1026 02:56:31.976235 (+ 31us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.976344 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068809}
I20251026 02:56:31.976624 2375 tablet_bootstrap.cc:492] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:31.976758 2376 tablet_replica.cc:333] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:31.976775 2375 ts_tablet_manager.cc:1403] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20251026 02:56:31.976962 2376 raft_consensus.cc:2243] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:31.977139 2376 raft_consensus.cc:2272] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:31.977658 2376 ts_tablet_manager.cc:1916] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:31.977753 2375 raft_consensus.cc:359] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:31.977965 2375 raft_consensus.cc:385] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:31.978061 2375 raft_consensus.cc:740] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:31.978363 2375 consensus_queue.cc:260] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:31.979154 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 1eabfc95caa84afc9a8956a7ac13fea6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1eabfc95caa84afc9a8956a7ac13fea6 already in progress: creating tablet
W20251026 02:56:31.979476 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 4b49c0996c4d41cbaddcdf0b75846bcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b49c0996c4d41cbaddcdf0b75846bcd already in progress: creating tablet
W20251026 02:56:31.979737 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7177931353164b2aa82c00b2807ccd63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7177931353164b2aa82c00b2807ccd63 already in progress: creating tablet
I20251026 02:56:31.982623 2375 ts_tablet_manager.cc:1434] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251026 02:56:31.982581 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5f363b68bc2c4e7685f25e4eefa532a8" } new_table_name: "default.Table_c02fc4c8f1994cabaee4a6aaf1874bdd" modify_external_catalogs: true
I20251026 02:56:31.982638 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ca53804528194849a18e026b5026f4b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:31.982926 2375 tablet_bootstrap.cc:492] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:31.984670 2375 tablet_bootstrap.cc:654] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:31.986948 3006 consensus_queue.cc:1048] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:31.973937 2378 tablet_bootstrap.cc:492] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:31.990105 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1a1b8c440061429caea0f0803f110742 consensus state has no leader
I20251026 02:56:31.990669 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c5ded183cc2844e98b226ab83673cea0" } new_table_name: "default.Table_632f764fc25e4762ad2de1c2a34cf713" modify_external_catalogs: true
I20251026 02:56:31.994542 2378 tablet_bootstrap.cc:654] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:31.995973 2053 consensus_peers.cc:597] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:31.997061 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6da5ba99f77e43cc93a15171e7355167"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.004622 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5824831b8e184daf8b40543686c06467 has no consensus state
I20251026 02:56:32.005239 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6b7a7d49a4cf4c7084d664fa624c3da4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:32.012331 2934 raft_consensus.cc:515] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.013146 2934 leader_election.cc:290] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.013608 2377 ts_tablet_manager.cc:1929] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:32.013751 2377 log.cc:1199] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/f8533f1c191042ee9f2c0b99972fe931
I20251026 02:56:32.014150 2377 ts_tablet_manager.cc:1950] T f8533f1c191042ee9f2c0b99972fe931 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:32.014437 3065 raft_consensus.cc:515] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.015100 3065 leader_election.cc:290] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.017051 2372 tablet_replica.cc:333] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:32.017443 2372 raft_consensus.cc:2243] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:32.017633 2372 raft_consensus.cc:2272] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:32.018216 2372 ts_tablet_manager.cc:1916] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:32.019295 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0efdcc36ff5b47379e0a76512e05d379" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:32.020463 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0efdcc36ff5b47379e0a76512e05d379" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:32.020722 2119 raft_consensus.cc:3060] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.020742 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 122de8616b8f4fcf88971d5738c47e82 (table default.Table_719b11bde46041c5a20750db14381807 [id=4912ce6ea2304b5a9c21dd91a7cbd56f]) successfully deleted
W20251026 02:56:32.021131 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9a08e8b7eb1949408f2151f2d7257e9d because the tablet was not found. No further retry: Not found: Tablet not found: 9a08e8b7eb1949408f2151f2d7257e9d
W20251026 02:56:32.021776 2194 leader_election.cc:343] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.022421 3096 tablet.cc:1722] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:32.023227 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94886ad6d6bb45f39261725169774ac9" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:32.023653 1976 raft_consensus.cc:3060] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:32.024121 2053 leader_election.cc:343] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.028281 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94886ad6d6bb45f39261725169774ac9" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:32.028541 2262 raft_consensus.cc:3060] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:32.032402 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca53804528194849a18e026b5026f4b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:32.032560 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b7a7d49a4cf4c7084d664fa624c3da4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.034519 2934 raft_consensus.cc:493] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.034694 2934 raft_consensus.cc:515] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.035094 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 369) took 4269 ms. Trace:
I20251026 02:56:32.035202 2377 rpcz_store.cc:276] 1026 02:56:27.765137 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.765202 (+ 65us) service_pool.cc:225] Handling call
1026 02:56:31.788064 (+4022862us) ts_tablet_manager.cc:1151] Deleting tablet f8533f1c191042ee9f2c0b99972fe931
1026 02:56:31.788075 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:31.913188 (+125113us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.013471 (+100283us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.035024 (+ 21553us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4022635}
I20251026 02:56:32.035392 2934 leader_election.cc:290] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.035583 2377 tablet_replica.cc:333] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:32.035789 2377 raft_consensus.cc:2243] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:32.036198 2377 raft_consensus.cc:2272] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:32.036777 2377 ts_tablet_manager.cc:1916] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:32.038975 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6da5ba99f77e43cc93a15171e7355167 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.046474 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet f8533f1c191042ee9f2c0b99972fe931 (table default.Table_e70cf6d4e0dc4017b484753f4b0250d9 [id=f4fb24ed75a24d81ae8f90619ded2ca9]) successfully deleted
I20251026 02:56:32.046916 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "049a915b35224cdaa38df77c734a0f01" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:32.047223 2263 raft_consensus.cc:2468] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:32.047808 2053 leader_election.cc:304] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:32.048187 2934 raft_consensus.cc:2804] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:32.048352 2934 raft_consensus.cc:493] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:32.048476 2934 raft_consensus.cc:3060] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.051759 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b0a34356c6e54ae79696bc5edfeb8aa1" } new_table_name: "default.Table_e75c4f87d9d54b479fba42c803c3bc9b" modify_external_catalogs: true
I20251026 02:56:32.053706 2381 tablet_bootstrap.cc:492] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:32.053660 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "049a915b35224cdaa38df77c734a0f01" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:32.054430 2053 leader_election.cc:343] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.055739 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d1d2850518cc4c51a5cd3296d2d19680"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.056425 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1d2850518cc4c51a5cd3296d2d19680 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.057729 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_dd6ea2361b074225af91ade940fbf0d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:32.058162 2375 tablet_bootstrap.cc:492] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:32.058142 3030 raft_consensus.cc:493] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:32.058276 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd6ea2361b074225af91ade940fbf0d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.058372 2375 ts_tablet_manager.cc:1403] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.075s user 0.014s sys 0.006s
I20251026 02:56:32.058322 3030 raft_consensus.cc:515] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.058933 3030 leader_election.cc:290] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.059278 2375 raft_consensus.cc:359] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.059573 2375 raft_consensus.cc:385] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.059659 2375 raft_consensus.cc:740] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.059897 2375 consensus_queue.cc:260] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:32.060289 1916 leader_election.cc:343] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9156314f98ac4ed9981f6f13fc585043
I20251026 02:56:32.063323 2375 ts_tablet_manager.cc:1434] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251026 02:56:32.063663 2375 tablet_bootstrap.cc:492] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:32.065505 2375 tablet_bootstrap.cc:654] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.080279 2381 ts_tablet_manager.cc:1403] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.179s user 0.034s sys 0.030s
I20251026 02:56:32.082383 2381 raft_consensus.cc:359] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.082733 2381 raft_consensus.cc:385] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.082851 2381 raft_consensus.cc:740] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.083555 2381 consensus_queue.cc:260] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.084599 2381 ts_tablet_manager.cc:1434] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251026 02:56:32.085395 2381 tablet_bootstrap.cc:492] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:32.087217 2195 consensus_peers.cc:597] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86d8e1e21c5f440d8baf76c357d01ad2. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:32.089576 2375 tablet_bootstrap.cc:492] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:32.089744 2375 ts_tablet_manager.cc:1403] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.020s
I20251026 02:56:32.090188 2381 tablet_bootstrap.cc:654] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.090654 2375 raft_consensus.cc:359] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.090838 2375 raft_consensus.cc:385] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.090936 2375 raft_consensus.cc:740] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.091181 2375 consensus_queue.cc:260] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.091696 2375 ts_tablet_manager.cc:1434] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:32.099906 2375 tablet_bootstrap.cc:492] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:32.101274 1915 leader_election.cc:343] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 9156314f98ac4ed9981f6f13fc585043
I20251026 02:56:32.101441 1915 leader_election.cc:304] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:32.101861 3030 raft_consensus.cc:2749] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:32.104671 2375 tablet_bootstrap.cc:654] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.107283 2378 tablet_bootstrap.cc:492] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:32.107656 2378 ts_tablet_manager.cc:1403] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.134s user 0.002s sys 0.010s
I20251026 02:56:32.108486 2378 raft_consensus.cc:359] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.108747 2378 raft_consensus.cc:385] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.108870 2378 raft_consensus.cc:740] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.109162 2378 consensus_queue.cc:260] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.112422 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet a821ed8a686c452a975acb96b5c74778 (table default.Table_016812daf097412081aff793ad75f72a [id=347d712aca4e477c89805846800fb824]). Expected version 2 got 0
I20251026 02:56:32.112988 2262 raft_consensus.cc:2468] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:32.113284 2378 ts_tablet_manager.cc:1434] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251026 02:56:32.113652 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_766ba994f1b5469cb050efb60fef1c42" } new_table_name: "default.Table_7ece4924086a4ae68db59650193f0e0e" modify_external_catalogs: true
I20251026 02:56:32.115183 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_679529c2b6914f4bbcba4a245a8cbb08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.115758 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_679529c2b6914f4bbcba4a245a8cbb08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.116240 2378 tablet_bootstrap.cc:492] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:32.116672 2376 ts_tablet_manager.cc:1929] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:32.116817 2376 log.cc:1199] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/155a23a93358487a9564bb5ac175b4eb
I20251026 02:56:32.116771 2114 raft_consensus.cc:2468] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:32.117192 2376 ts_tablet_manager.cc:1950] T 155a23a93358487a9564bb5ac175b4eb P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:32.121610 2378 tablet_bootstrap.cc:654] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:32.124595 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ea364c4a5cf4f3f99d9ff6904add697 consensus state has no leader
I20251026 02:56:32.125775 2375 tablet_bootstrap.cc:492] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:32.125928 2375 ts_tablet_manager.cc:1403] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20251026 02:56:32.126645 2375 raft_consensus.cc:359] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.126843 2375 raft_consensus.cc:385] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.126900 2375 raft_consensus.cc:740] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.127151 2375 consensus_queue.cc:260] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.127637 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_98bf1a1deafd476e85a4d04987d9355d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.128180 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98bf1a1deafd476e85a4d04987d9355d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.129788 1657 catalog_manager.cc:5649] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:32.131052 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 94886ad6d6bb45f39261725169774ac9 (table default.Table_d23ac2c9657b4adeaac9545d5bc9ae41 [id=9e897d33e6374ad4bc3c8acaae512645]). Expected version 1 got 0
I20251026 02:56:32.131343 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 358e805370ef410e9d65ee92a8d04f34 (table default.Table_71583d73a49142ae9fec31193fb53542 [id=e86d9fc892a8409590c555e25b27a12f]). Expected version 1 got 0
I20251026 02:56:32.132390 2375 ts_tablet_manager.cc:1434] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251026 02:56:32.132735 2375 tablet_bootstrap.cc:492] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:32.134943 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 4e645650709f47eea4243ab64ec31312 (DEFAULT_TABLE table=default.Table_e87acf02545f4f63987389f32903aec1 [id=2688acec03214505b27b9f70fda3c6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.135429 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 5824831b8e184daf8b40543686c06467 (DEFAULT_TABLE table=default.Table_632f764fc25e4762ad2de1c2a34cf713 [id=2ee048e6370c4e4fb996f80c4304da80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.139535 2375 tablet_bootstrap.cc:654] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.143882 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e645650709f47eea4243ab64ec31312. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.144560 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5824831b8e184daf8b40543686c06467. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.156545 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 13acb028b89e47eeadff91fe61cbebf3 (DEFAULT_TABLE table=default.Table_857acd1afff5412b8945e40b98c1dde6 [id=f6b9a96f766f491fa6967954c38aaa2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.157092 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13acb028b89e47eeadff91fe61cbebf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.157238 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 13acb028b89e47eeadff91fe61cbebf3 (DEFAULT_TABLE table=default.Table_857acd1afff5412b8945e40b98c1dde6 [id=f6b9a96f766f491fa6967954c38aaa2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.159978 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 95a4aa0abc0b455186482b0a85178db9 (DEFAULT_TABLE table=default.Table_53d4276e57e646faa0ed4bfab8ae8e82 [id=b5d10993054047c48f63b65404567795]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.160466 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95a4aa0abc0b455186482b0a85178db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.166047 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_20e0d029dadf4384b1efbfbc372f4741" } new_table_name: "default.Table_7ea22608d74d41f4853bb29edc7e5c52" modify_external_catalogs: true
I20251026 02:56:32.167759 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f28a7b90302c4e3993516e647d2bc67c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.168300 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f28a7b90302c4e3993516e647d2bc67c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.168668 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9fb42b54f558499090a1d7cf257dd84c
I20251026 02:56:32.169464 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9fb0008346da44318ccc90cd69474750" } modify_external_catalogs: true
I20251026 02:56:32.169658 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9fb0008346da44318ccc90cd69474750" } modify_external_catalogs: true
I20251026 02:56:32.171061 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 21b936fa4f6a4ff6af321971a30cccbe
I20251026 02:56:32.171330 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fb42b54f558499090a1d7cf257dd84c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:32.171547 1661 catalog_manager.cc:6719] ReplaceTablet: tablet 21b936fa4f6a4ff6af321971a30cccbe deleted and replaced by tablet a956ebfb1eaa4d6fb1575b4165293193
I20251026 02:56:32.171792 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 5824831b8e184daf8b40543686c06467 (DEFAULT_TABLE table=default.Table_632f764fc25e4762ad2de1c2a34cf713 [id=2ee048e6370c4e4fb996f80c4304da80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.171978 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_051c1782e1cd4f8caa8dbefeddb868f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.172494 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_051c1782e1cd4f8caa8dbefeddb868f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.174162 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 027dd92fc5cf4c9e847de3e55e0dac1c
I20251026 02:56:32.174870 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_44853e936f824114a26a5c01ae8543d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.175742 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46547883cfe041f3954aa4128fba1d8c has no consensus state
I20251026 02:56:32.175884 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 21b936fa4f6a4ff6af321971a30cccbe with delete_type TABLET_DATA_DELETED (replaced by tablet a956ebfb1eaa4d6fb1575b4165293193) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:32.176049 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 86e483c52e514f1da54335479f59d68e from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:32.176512 2053 leader_election.cc:304] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.176962 3006 raft_consensus.cc:2804] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:32.177100 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 4a3f3fc3c7a04d05bae28472251f3acc
I20251026 02:56:32.177116 3006 raft_consensus.cc:697] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:32.177377 1949 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fb42b54f558499090a1d7cf257dd84c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.177379 3006 consensus_queue.cc:237] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.177966 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 9156314f98ac4ed9981f6f13fc585043 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.178275 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 027dd92fc5cf4c9e847de3e55e0dac1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:32.178413 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a3f3fc3c7a04d05bae28472251f3acc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:32.178498 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 4e645650709f47eea4243ab64ec31312 (DEFAULT_TABLE table=default.Table_e87acf02545f4f63987389f32903aec1 [id=2688acec03214505b27b9f70fda3c6c4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:32.180080 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44853e936f824114a26a5c01ae8543d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.180280 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet f8533f1c191042ee9f2c0b99972fe931 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.181056 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 21b936fa4f6a4ff6af321971a30cccbe with delete_type TABLET_DATA_DELETED (replaced by tablet a956ebfb1eaa4d6fb1575b4165293193) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.181241 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 95a4aa0abc0b455186482b0a85178db9 (DEFAULT_TABLE table=default.Table_53d4276e57e646faa0ed4bfab8ae8e82 [id=b5d10993054047c48f63b65404567795]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.182574 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a3f3fc3c7a04d05bae28472251f3acc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.185708 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 352a284902c94f088e2e6ae006162db0 (table default.Table_6cd4ee82e9ec406f971669b7f0722677 [id=414375003cd44b5c8c9f0249b339c28a]). Expected version 1 got 0
I20251026 02:56:32.186239 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 358e805370ef410e9d65ee92a8d04f34 (table default.Table_71583d73a49142ae9fec31193fb53542 [id=e86d9fc892a8409590c555e25b27a12f]). Expected version 1 got 0
I20251026 02:56:32.186921 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 63a63b796191465aab9a63d3ffb0554d
I20251026 02:56:32.188728 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 63a63b796191465aab9a63d3ffb0554d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:32.190938 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 4e645650709f47eea4243ab64ec31312 (DEFAULT_TABLE table=default.Table_e87acf02545f4f63987389f32903aec1 [id=2688acec03214505b27b9f70fda3c6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.191867 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 902cc4f9dbbd4d8baa829448f2a4687a (DEFAULT_TABLE table=default.Table_8ab273ba34e54a89910acbe0f5370372 [id=4d82f380fe32413fa5ec73cf0261d9b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.192032 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 5824831b8e184daf8b40543686c06467 (DEFAULT_TABLE table=default.Table_632f764fc25e4762ad2de1c2a34cf713 [id=2ee048e6370c4e4fb996f80c4304da80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.192180 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 13acb028b89e47eeadff91fe61cbebf3 (DEFAULT_TABLE table=default.Table_857acd1afff5412b8945e40b98c1dde6 [id=f6b9a96f766f491fa6967954c38aaa2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.192957 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 95a4aa0abc0b455186482b0a85178db9 (DEFAULT_TABLE table=default.Table_53d4276e57e646faa0ed4bfab8ae8e82 [id=b5d10993054047c48f63b65404567795]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.193145 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 027dd92fc5cf4c9e847de3e55e0dac1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.193339 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fb42b54f558499090a1d7cf257dd84c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:32.193392 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 63a63b796191465aab9a63d3ffb0554d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.193405 2195 leader_election.cc:304] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.193552 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 21b936fa4f6a4ff6af321971a30cccbe with delete_type TABLET_DATA_DELETED (replaced by tablet a956ebfb1eaa4d6fb1575b4165293193) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:32.193806 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 027dd92fc5cf4c9e847de3e55e0dac1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:32.193969 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a3f3fc3c7a04d05bae28472251f3acc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:32.194166 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 63a63b796191465aab9a63d3ffb0554d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:32.194096 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b4f29de6c177447f81779bfa9e30ac2d" } new_table_name: "default.Table_30611e161c274e8b99bd1251e5e527aa" modify_external_catalogs: true
I20251026 02:56:32.194373 3065 raft_consensus.cc:2804] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:32.194525 3065 raft_consensus.cc:697] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:32.194787 3065 consensus_queue.cc:237] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.197801 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 902cc4f9dbbd4d8baa829448f2a4687a (DEFAULT_TABLE table=default.Table_8ab273ba34e54a89910acbe0f5370372 [id=4d82f380fe32413fa5ec73cf0261d9b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.200349 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 7961528688cb41e5be1ba380f954f66a (DEFAULT_TABLE table=default.Table_e66bf47ad1824afeafd5c77400d56044 [id=d50425d46cb14e8d9a1707aff7e446af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.200555 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 7961528688cb41e5be1ba380f954f66a (DEFAULT_TABLE table=default.Table_e66bf47ad1824afeafd5c77400d56044 [id=d50425d46cb14e8d9a1707aff7e446af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.201956 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 902cc4f9dbbd4d8baa829448f2a4687a (DEFAULT_TABLE table=default.Table_8ab273ba34e54a89910acbe0f5370372 [id=4d82f380fe32413fa5ec73cf0261d9b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.203701 2378 tablet_bootstrap.cc:492] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:32.205284 2378 ts_tablet_manager.cc:1403] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.089s user 0.017s sys 0.010s
I20251026 02:56:32.207013 2378 raft_consensus.cc:359] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.207667 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_19637f030638473d8f906e789a7cdb01" } new_table_name: "default.Table_436964d7d7d54f38a72eebf1d65d8d6e" modify_external_catalogs: true
I20251026 02:56:32.208456 2934 raft_consensus.cc:515] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.209223 2934 leader_election.cc:290] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.209618 2119 raft_consensus.cc:2468] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:32.209096 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 463) took 1299 ms. Trace:
I20251026 02:56:32.210214 2378 raft_consensus.cc:385] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.210106 2376 rpcz_store.cc:276] 1026 02:56:30.909835 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.909895 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:31.976702 (+1066807us) ts_tablet_manager.cc:1151] Deleting tablet 155a23a93358487a9564bb5ac175b4eb
1026 02:56:31.976710 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.016269 (+ 39559us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.116550 (+100281us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.209041 (+ 92491us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1066582}
I20251026 02:56:32.210641 2378 raft_consensus.cc:740] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.210886 2376 tablet_replica.cc:333] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:32.211376 2376 raft_consensus.cc:2243] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:32.211594 2376 raft_consensus.cc:2272] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:32.211723 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e645650709f47eea4243ab64ec31312. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.212145 2376 ts_tablet_manager.cc:1916] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:32.209338 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5824831b8e184daf8b40543686c06467. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.212402 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13acb028b89e47eeadff91fe61cbebf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.212721 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "049a915b35224cdaa38df77c734a0f01" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:32.212847 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5824831b8e184daf8b40543686c06467. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.213349 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95a4aa0abc0b455186482b0a85178db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.213827 2377 ts_tablet_manager.cc:1929] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:32.213932 2377 log.cc:1199] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/78a624d839934d3089b64e397d4360af
I20251026 02:56:32.214357 2377 ts_tablet_manager.cc:1950] T 78a624d839934d3089b64e397d4360af P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:32.214629 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 902cc4f9dbbd4d8baa829448f2a4687a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.215832 2378 consensus_queue.cc:260] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.216361 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e645650709f47eea4243ab64ec31312. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.217368 2378 ts_tablet_manager.cc:1434] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251026 02:56:32.219627 2378 tablet_bootstrap.cc:492] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:32.220387 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95a4aa0abc0b455186482b0a85178db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.221886 2378 tablet_bootstrap.cc:654] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:32.223475 2053 leader_election.cc:343] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.226007 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13acb028b89e47eeadff91fe61cbebf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.227214 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 902cc4f9dbbd4d8baa829448f2a4687a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.229418 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7961528688cb41e5be1ba380f954f66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.233237 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 155a23a93358487a9564bb5ac175b4eb (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]) successfully deleted
I20251026 02:56:32.242051 2195 leader_election.cc:304] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.242197 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "049a915b35224cdaa38df77c734a0f01" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:32.242447 2262 raft_consensus.cc:3060] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.245328 3065 raft_consensus.cc:2804] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:32.245496 3065 raft_consensus.cc:697] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:32.245750 3065 consensus_queue.cc:237] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.247107 2934 raft_consensus.cc:493] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.247632 2934 raft_consensus.cc:515] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.248394 2934 leader_election.cc:290] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.251190 3028 raft_consensus.cc:493] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.251368 3028 raft_consensus.cc:515] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.252048 3028 leader_election.cc:290] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.252419 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "634d913c98c842a49d97a54f23752d26" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:32.253108 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2bfd9b55c484ef8a9c6345d56f794fb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:32.253183 2053 leader_election.cc:343] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:32.253700 2194 leader_election.cc:343] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.258579 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_082dd939577943e5ae1d59dea699d4ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.259366 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_082dd939577943e5ae1d59dea699d4ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.264189 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 902cc4f9dbbd4d8baa829448f2a4687a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.265666 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7961528688cb41e5be1ba380f954f66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.270110 3065 raft_consensus.cc:493] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.270398 3065 raft_consensus.cc:515] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.271929 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "122de8616b8f4fcf88971d5738c47e82" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:32.271972 3065 leader_election.cc:290] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:32.272514 2194 leader_election.cc:343] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.272202 2378 tablet_bootstrap.cc:492] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:32.272778 2378 ts_tablet_manager.cc:1403] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.002s
I20251026 02:56:32.273805 2378 raft_consensus.cc:359] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.274055 2378 raft_consensus.cc:385] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.274163 2378 raft_consensus.cc:740] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.274550 2378 consensus_queue.cc:260] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.275377 2378 ts_tablet_manager.cc:1434] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:32.275786 2378 tablet_bootstrap.cc:492] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:32.279328 2372 ts_tablet_manager.cc:1929] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:32.279515 2372 log.cc:1199] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/ced7ad6a601d432cbc13b3199a0506e4
I20251026 02:56:32.279892 2372 ts_tablet_manager.cc:1950] T ced7ad6a601d432cbc13b3199a0506e4 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:32.283151 2378 tablet_bootstrap.cc:654] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.289276 3096 tablet.cc:1722] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:32.289574 2378 tablet_bootstrap.cc:492] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:32.289781 2378 ts_tablet_manager.cc:1403] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251026 02:56:32.293538 2378 raft_consensus.cc:359] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.293766 2378 raft_consensus.cc:385] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.293843 2378 raft_consensus.cc:740] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.294096 2378 consensus_queue.cc:260] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:32.295022 2194 consensus_peers.cc:597] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:32.295708 2378 ts_tablet_manager.cc:1434] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251026 02:56:32.296494 2378 tablet_bootstrap.cc:492] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:32.297322 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 370) took 4531 ms. Trace:
I20251026 02:56:32.297492 2377 rpcz_store.cc:276] 1026 02:56:27.765568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.765612 (+ 44us) service_pool.cc:225] Handling call
1026 02:56:32.035487 (+4269875us) ts_tablet_manager.cc:1151] Deleting tablet 78a624d839934d3089b64e397d4360af
1026 02:56:32.035498 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.115797 (+ 80299us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.213703 (+ 97906us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.297281 (+ 83578us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4269595}
I20251026 02:56:32.298774 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 371) took 4509 ms. Trace:
I20251026 02:56:32.298929 2377 rpcz_store.cc:276] 1026 02:56:27.789695 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.789749 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:32.298637 (+4508888us) ts_tablet_manager.cc:1151] Deleting tablet f437d8b5f5784e8fbec10d0700e90348
1026 02:56:32.298648 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.298744 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4508375}
I20251026 02:56:32.300166 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 379) took 4426 ms. Trace:
I20251026 02:56:32.300331 2377 rpcz_store.cc:276] 1026 02:56:27.873778 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:27.874699 (+ 921us) service_pool.cc:225] Handling call
1026 02:56:32.300010 (+4425311us) ts_tablet_manager.cc:1151] Deleting tablet 9c7efb70c0804c149f36440a4a9e6ef8
1026 02:56:32.300031 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.300140 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4425124}
I20251026 02:56:32.300719 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 386) took 4201 ms. Trace:
I20251026 02:56:32.300865 2377 rpcz_store.cc:276] 1026 02:56:28.098815 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.099093 (+ 278us) service_pool.cc:225] Handling call
1026 02:56:32.300612 (+4201519us) ts_tablet_manager.cc:1151] Deleting tablet 86e483c52e514f1da54335479f59d68e
1026 02:56:32.300620 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.300694 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4201217}
I20251026 02:56:32.301173 2377 tablet_replica.cc:333] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:32.301409 2377 raft_consensus.cc:2243] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:32.301606 2377 raft_consensus.cc:2272] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:32.304811 2378 tablet_bootstrap.cc:654] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.307034 2377 ts_tablet_manager.cc:1916] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:32.308458 2262 raft_consensus.cc:2468] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:32.309134 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 494) took 1183 ms. Trace:
I20251026 02:56:32.309345 2372 rpcz_store.cc:276] 1026 02:56:31.125484 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.126196 (+ 712us) service_pool.cc:225] Handling call
1026 02:56:32.016977 (+890781us) ts_tablet_manager.cc:1151] Deleting tablet ced7ad6a601d432cbc13b3199a0506e4
1026 02:56:32.016984 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.148515 (+131531us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.279182 (+130667us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.309064 (+ 29882us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":890603}
I20251026 02:56:32.310359 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet ced7ad6a601d432cbc13b3199a0506e4 (table default.Table_241789d6111d497c89efdb4507dff32e [id=4e7fddec9b124b32b7786ea15111878b]) successfully deleted
I20251026 02:56:32.310971 2372 tablet_replica.cc:333] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:32.311568 2372 raft_consensus.cc:2243] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:32.312121 2372 raft_consensus.cc:2272] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:32.313356 2372 ts_tablet_manager.cc:1916] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:32.315249 1950 tablet_service.cc:1505] Processing CreateTablet for tablet d4d64902618d46428ff345bc190b03bd (DEFAULT_TABLE table=default.Table_7af610f8acba426b84a2c85d1d4f1c5c [id=76cace938e21458bacf48c76e3f570d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.321621 2378 tablet_bootstrap.cc:492] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:32.321795 2378 ts_tablet_manager.cc:1403] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.005s
I20251026 02:56:32.322621 2378 raft_consensus.cc:359] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.322816 2378 raft_consensus.cc:385] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.322904 2378 raft_consensus.cc:740] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.323153 2378 consensus_queue.cc:260] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.323493 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2bfd9b55c484ef8a9c6345d56f794fb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:32.323694 2119 raft_consensus.cc:2468] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:32.323757 2378 ts_tablet_manager.cc:1434] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:32.324047 2378 tablet_bootstrap.cc:492] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:32.324525 2086 tablet_service.cc:1505] Processing CreateTablet for tablet d4d64902618d46428ff345bc190b03bd (DEFAULT_TABLE table=default.Table_7af610f8acba426b84a2c85d1d4f1c5c [id=76cace938e21458bacf48c76e3f570d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.325032 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d64902618d46428ff345bc190b03bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:32.328891 2195 consensus_peers.cc:597] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98fe44aab33b40eba819386c3bffca46. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:32.328857 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "634d913c98c842a49d97a54f23752d26" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:32.329295 2195 leader_election.cc:304] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.329668 2962 raft_consensus.cc:2804] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:32.329814 2962 raft_consensus.cc:493] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:32.329911 2962 raft_consensus.cc:3060] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.334739 2378 tablet_bootstrap.cc:654] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.370285 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 (DEFAULT_TABLE table=default.Table_60d710dc906a41308e183f47acdda437 [id=153672f9c43d44549e94d94e4e978f73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.370745 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 (DEFAULT_TABLE table=default.Table_60d710dc906a41308e183f47acdda437 [id=153672f9c43d44549e94d94e4e978f73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.371224 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.371932 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 163a71c4f1c9488a8cc442f061ed967f (DEFAULT_TABLE table=default.Table_171b77d4c8cb4fb48db2cb541d40d870 [id=5cffa817c8294a0caeb299b8bf8aad7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.372985 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 163a71c4f1c9488a8cc442f061ed967f (DEFAULT_TABLE table=default.Table_171b77d4c8cb4fb48db2cb541d40d870 [id=5cffa817c8294a0caeb299b8bf8aad7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.373462 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163a71c4f1c9488a8cc442f061ed967f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:32.379766 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 50469.7us
W20251026 02:56:32.380141 2195 leader_election.cc:343] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 122de8616b8f4fcf88971d5738c47e82
I20251026 02:56:32.380486 2195 leader_election.cc:304] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:32.380740 2195 raft_consensus.cc:2686] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251026 02:56:32.381265 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 78a624d839934d3089b64e397d4360af (table default.Table_a72502f837fb4542b1f00bab5da37724 [id=4db9b61c88b5422fade5d42038bc5bcf]) successfully deleted
I20251026 02:56:32.381428 2053 leader_election.cc:304] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:32.381542 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet f437d8b5f5784e8fbec10d0700e90348 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f437d8b5f5784e8fbec10d0700e90348 already in progress: creating tablet
W20251026 02:56:32.381758 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 9c7efb70c0804c149f36440a4a9e6ef8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c7efb70c0804c149f36440a4a9e6ef8 already in progress: creating tablet
W20251026 02:56:32.381968 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 86e483c52e514f1da54335479f59d68e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86e483c52e514f1da54335479f59d68e already in progress: creating tablet
W20251026 02:56:32.383538 2053 leader_election.cc:343] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.383700 2053 leader_election.cc:304] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.384759 2934 raft_consensus.cc:2804] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:32.384907 2934 raft_consensus.cc:697] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:32.384954 3007 raft_consensus.cc:2749] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:32.385216 2934 consensus_queue.cc:237] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:32.390913 2194 consensus_peers.cc:597] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:32.405272 3006 raft_consensus.cc:493] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.405560 3006 raft_consensus.cc:515] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.406884 3006 leader_election.cc:290] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.409225 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5caeb11322fe48348ebb523967152187" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:32.409906 2053 leader_election.cc:343] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.411060 2962 raft_consensus.cc:515] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.411711 2962 leader_election.cc:290] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.412524 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2bfd9b55c484ef8a9c6345d56f794fb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:32.413261 2194 leader_election.cc:343] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.416594 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 5b8e1d79d2a247dba25540daf6b81bf0 (DEFAULT_TABLE table=default.Table_0ea323ca655f45aa94b47eab713ebbd0 [id=1d82a5d03031489fa756df9cfb380b8e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:32.421353 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1406c22ee4ee4f94acecad9546208e79 consensus state has no leader
I20251026 02:56:32.422943 1663 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 903e65fa8c214dc9a50cb97cab101d22 (table default.Table_2d168c00cb884360bdb2475bd391f576 [id=65b5770c4b9243b892bfab963a439bb9]). Expected version 4 got 0
I20251026 02:56:32.423223 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3791cb18fe1f4937badb4859d00647f3
I20251026 02:56:32.423710 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8929007b52484dcca450eb9ea54fe75d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:32.423887 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f5e1fc5d0a604a82be231f8b70013315"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.424345 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5e1fc5d0a604a82be231f8b70013315 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.424893 1946 tablet_service.cc:1505] Processing CreateTablet for tablet a74d6a4e041347578b0976a8b54bf22c (DEFAULT_TABLE table=default.Table_8cfa87d9be80442b941db99ddc6021d2 [id=d9e92619fc9d43dca6da2fad8608cb5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.425361 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a74d6a4e041347578b0976a8b54bf22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.425920 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d64902618d46428ff345bc190b03bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.426311 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.426685 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163a71c4f1c9488a8cc442f061ed967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.426632 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_bc49906123d84d55a0849db64bb7c25a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.427165 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc49906123d84d55a0849db64bb7c25a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.427876 2376 ts_tablet_manager.cc:1929] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:32.427982 2376 log.cc:1199] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/4a5343ba771a4e8ea2f467c4c619c1f9
I20251026 02:56:32.428313 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 3791cb18fe1f4937badb4859d00647f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:32.428416 2376 ts_tablet_manager.cc:1950] T 4a5343ba771a4e8ea2f467c4c619c1f9 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:32.428942 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5edc3cfde43b432ca349a34eb81cc365"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:32.429410 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b8e1d79d2a247dba25540daf6b81bf0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:32.429486 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5edc3cfde43b432ca349a34eb81cc365 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:32.424319 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8929007b52484dcca450eb9ea54fe75d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.435577 1952 tablet_service.cc:1552] Processing DeleteTablet for tablet 3791cb18fe1f4937badb4859d00647f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.436975 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 5b8e1d79d2a247dba25540daf6b81bf0 (DEFAULT_TABLE table=default.Table_0ea323ca655f45aa94b47eab713ebbd0 [id=1d82a5d03031489fa756df9cfb380b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.438027 2962 raft_consensus.cc:493] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.438210 2962 raft_consensus.cc:515] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.438477 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b8e1d79d2a247dba25540daf6b81bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.439599 2962 leader_election.cc:290] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.440025 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6d6f024d8b943d5acf69b64bc7c7214" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:32.440956 2194 leader_election.cc:343] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.442034 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 7961528688cb41e5be1ba380f954f66a (DEFAULT_TABLE table=default.Table_e66bf47ad1824afeafd5c77400d56044 [id=d50425d46cb14e8d9a1707aff7e446af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.442622 2233 tablet_service.cc:1505] Processing CreateTablet for tablet d4d64902618d46428ff345bc190b03bd (DEFAULT_TABLE table=default.Table_7af610f8acba426b84a2c85d1d4f1c5c [id=76cace938e21458bacf48c76e3f570d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.443046 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 (DEFAULT_TABLE table=default.Table_60d710dc906a41308e183f47acdda437 [id=153672f9c43d44549e94d94e4e978f73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.443130 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2bfd9b55c484ef8a9c6345d56f794fb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:32.443379 2119 raft_consensus.cc:3060] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.444432 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5caeb11322fe48348ebb523967152187" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:32.448081 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6d6f024d8b943d5acf69b64bc7c7214" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:32.448364 2114 raft_consensus.cc:2468] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:32.448506 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 163a71c4f1c9488a8cc442f061ed967f (DEFAULT_TABLE table=default.Table_171b77d4c8cb4fb48db2cb541d40d870 [id=5cffa817c8294a0caeb299b8bf8aad7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.448907 2962 raft_consensus.cc:493] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.448969 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 5b8e1d79d2a247dba25540daf6b81bf0 (DEFAULT_TABLE table=default.Table_0ea323ca655f45aa94b47eab713ebbd0 [id=1d82a5d03031489fa756df9cfb380b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.449028 2962 raft_consensus.cc:515] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.449357 2226 tablet_service.cc:1505] Processing CreateTablet for tablet a74d6a4e041347578b0976a8b54bf22c (DEFAULT_TABLE table=default.Table_8cfa87d9be80442b941db99ddc6021d2 [id=d9e92619fc9d43dca6da2fad8608cb5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.449661 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 3791cb18fe1f4937badb4859d00647f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:32.450063 2962 leader_election.cc:290] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.450409 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ced7ad6a601d432cbc13b3199a0506e4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:32.451066 2194 leader_election.cc:343] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.452541 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7961528688cb41e5be1ba380f954f66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.453189 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a74d6a4e041347578b0976a8b54bf22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.453625 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b8e1d79d2a247dba25540daf6b81bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.454069 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d64902618d46428ff345bc190b03bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.454490 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163a71c4f1c9488a8cc442f061ed967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.455930 2091 tablet_service.cc:1505] Processing CreateTablet for tablet a74d6a4e041347578b0976a8b54bf22c (DEFAULT_TABLE table=default.Table_8cfa87d9be80442b941db99ddc6021d2 [id=d9e92619fc9d43dca6da2fad8608cb5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.456367 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a74d6a4e041347578b0976a8b54bf22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.465184 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:32.480923 1650 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 049a915b35224cdaa38df77c734a0f01 (table default.Table_1818605e0bc548938337de5d5fc7a781 [id=f1f9d2ad39b34abbbe989bc3fac62b6c], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 049a915b35224cdaa38df77c734a0f01 consensus state has no leader
I20251026 02:56:32.481966 2962 raft_consensus.cc:493] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.482184 2962 raft_consensus.cc:515] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.483619 3065 raft_consensus.cc:493] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.483657 3030 raft_consensus.cc:493] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.483743 3065 raft_consensus.cc:515] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.483888 3030 raft_consensus.cc:515] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.484331 3065 leader_election.cc:290] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.484704 3030 leader_election.cc:290] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.485159 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e56b259a314422aba35eb129021bbeb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:32.485563 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47bfff40c181432896d6898ea50cd025" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:32.485716 2194 leader_election.cc:343] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.485965 2962 leader_election.cc:290] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.492115 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa399e6407fd4c4d9871d3d5ccf894ec" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:32.492488 1976 raft_consensus.cc:2468] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:32.492940 2194 leader_election.cc:304] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:32.493348 2962 raft_consensus.cc:2804] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:32.493468 2962 raft_consensus.cc:493] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:32.493575 2962 raft_consensus.cc:3060] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.500656 2381 tablet_bootstrap.cc:492] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:32.500818 2381 ts_tablet_manager.cc:1403] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.416s user 0.020s sys 0.028s
I20251026 02:56:32.501606 2381 raft_consensus.cc:359] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.501802 2381 raft_consensus.cc:385] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.501858 2381 raft_consensus.cc:740] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.502097 2381 consensus_queue.cc:260] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.502617 2381 ts_tablet_manager.cc:1434] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:32.502915 2381 tablet_bootstrap.cc:492] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:32.503559 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 464) took 1593 ms. Trace:
I20251026 02:56:32.503708 2376 rpcz_store.cc:276] 1026 02:56:30.910026 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.910157 (+ 131us) service_pool.cc:225] Handling call
1026 02:56:32.210790 (+1300633us) ts_tablet_manager.cc:1151] Deleting tablet 4a5343ba771a4e8ea2f467c4c619c1f9
1026 02:56:32.210800 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.312157 (+101357us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.427746 (+115589us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.503499 (+ 75753us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1300445}
I20251026 02:56:32.504130 2376 tablet_replica.cc:333] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:32.504376 2376 raft_consensus.cc:2243] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:32.504537 2381 tablet_bootstrap.cc:654] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.504558 2376 raft_consensus.cc:2272] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:32.505129 2376 ts_tablet_manager.cc:1916] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:32.506217 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 4a5343ba771a4e8ea2f467c4c619c1f9 (table default.Table_7296727c5b2f4c95ac3ff62818bd84f2 [id=af7a0605122745588d028814d3583cc6]) successfully deleted
W20251026 02:56:32.506790 2802 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:34768 (request call id 0) took 815 ms (client timeout 1000 ms). Trace:
W20251026 02:56:32.506937 2802 rpcz_store.cc:269] 1026 02:56:31.691305 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.691363 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:32.506736 (+815373us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 02:56:32.507683 1624 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu pid 2485: Timed out: CheckLeaks RPC to 127.1.150.60:33729 timed out after 1.000s (SENT)
I20251026 02:56:32.507829 1624 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu with pid 2485
W20251026 02:56:32.514796 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7abcdac4a70b48ad9fbba1bc8a60283b consensus state has no leader
I20251026 02:56:32.515067 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet a3af2695c41a4439bc4092b7cb1b2edf from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:32.515627 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ca53804528194849a18e026b5026f4b9" } new_table_name: "default.Table_cef54300186c47eca175dff1adf385db" modify_external_catalogs: true
I20251026 02:56:32.520427 2381 tablet_bootstrap.cc:492] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:32.520577 2381 ts_tablet_manager.cc:1403] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20251026 02:56:32.521389 2381 raft_consensus.cc:359] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.521559 2381 raft_consensus.cc:385] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.521606 2381 raft_consensus.cc:740] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.521806 2381 consensus_queue.cc:260] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.522403 2381 ts_tablet_manager.cc:1434] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:32.522452 1657 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_2d168c00cb884360bdb2475bd391f576 [id=65b5770c4b9243b892bfab963a439bb9] alter complete (version 4)
I20251026 02:56:32.522712 2381 tablet_bootstrap.cc:492] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:32.523711 1657 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.3:36919 (request call id 80) took 863 ms (client timeout 1000 ms). Trace:
W20251026 02:56:32.523826 1657 rpcz_store.cc:269] 1026 02:56:31.660176 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.753640 (+ 93464us) service_pool.cc:225] Handling call
1026 02:56:32.523648 (+770008us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:32.132556 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:32.132634 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:32.132638 (+ 4us) write_op.cc:435] Acquired schema lock
1026 02:56:32.132640 (+ 2us) tablet.cc:660] Decoding operations
1026 02:56:32.132724 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:32.132735 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:32.132741 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1026 02:56:32.132862 (+ 121us) tablet.cc:699] Row locks acquired
1026 02:56:32.132866 (+ 4us) write_op.cc:260] PREPARE: finished
1026 02:56:32.132893 (+ 27us) write_op.cc:270] Start()
1026 02:56:32.132915 (+ 22us) write_op.cc:276] Timestamp: P: 1761447392132829 usec, L: 0
1026 02:56:32.132918 (+ 3us) op_driver.cc:348] REPLICATION: starting
1026 02:56:32.133041 (+ 123us) log.cc:844] Serialized 527 byte log entry
1026 02:56:32.163284 (+ 30243us) op_driver.cc:464] REPLICATION: finished
1026 02:56:32.166465 (+ 3181us) write_op.cc:301] APPLY: starting
1026 02:56:32.166474 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:32.166498 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:32.166540 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:32.166573 (+ 33us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:32.166595 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:32.166600 (+ 5us) write_op.cc:312] APPLY: finished
1026 02:56:32.166648 (+ 48us) log.cc:844] Serialized 23 byte log entry
1026 02:56:32.166684 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:32.166695 (+ 11us) write_op.cc:454] Released schema lock
1026 02:56:32.166700 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:32.166761 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:32.166764 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
1026 02:56:32.204332 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:32.204978 (+ 646us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:32.204987 (+ 9us) write_op.cc:435] Acquired schema lock
1026 02:56:32.204991 (+ 4us) tablet.cc:660] Decoding operations
1026 02:56:32.205820 (+ 829us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:32.205839 (+ 19us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:32.205943 (+ 104us) tablet.cc:683] Acquiring locks for 1 operations
1026 02:56:32.206361 (+ 418us) tablet.cc:699] Row locks acquired
1026 02:56:32.206365 (+ 4us) write_op.cc:260] PREPARE: finished
1026 02:56:32.316696 (+110331us) write_op.cc:270] Start()
1026 02:56:32.316748 (+ 52us) write_op.cc:276] Timestamp: P: 1761447392316640 usec, L: 0
1026 02:56:32.316753 (+ 5us) op_driver.cc:348] REPLICATION: starting
1026 02:56:32.316975 (+ 222us) log.cc:844] Serialized 356 byte log entry
1026 02:56:32.511343 (+194368us) op_driver.cc:464] REPLICATION: finished
1026 02:56:32.512097 (+ 754us) write_op.cc:301] APPLY: starting
1026 02:56:32.512116 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:32.512164 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:32.512194 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:32.512316 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:32.512354 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:32.512363 (+ 9us) write_op.cc:312] APPLY: finished
1026 02:56:32.512470 (+ 107us) log.cc:844] Serialized 23 byte log entry
1026 02:56:32.512544 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:32.512566 (+ 22us) write_op.cc:454] Released schema lock
1026 02:56:32.512573 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:32.512649 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:32.512653 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":3145,"apply.run_cpu_time_us":294,"apply.run_wall_time_us":293,"num_ops":1,"prepare.queue_time_us":703,"prepare.run_cpu_time_us":464,"prepare.run_wall_time_us":528,"replication_time_us":30352}],["op",{"apply.queue_time_us":696,"apply.run_cpu_time_us":544,"apply.run_wall_time_us":542,"num_ops":1,"prepare.queue_time_us":19281,"prepare.run_cpu_time_us":1002,"prepare.run_wall_time_us":113260,"replication_time_us":194543}]]}
I20251026 02:56:32.524837 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet 9156314f98ac4ed9981f6f13fc585043 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:32.525210 2381 tablet_bootstrap.cc:654] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:32.526521 1660 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.2:34337 (request call id 99) took 872 ms (client timeout 1000 ms). Trace:
W20251026 02:56:32.526664 1660 rpcz_store.cc:269] 1026 02:56:31.654490 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.751345 (+ 96855us) service_pool.cc:225] Handling call
1026 02:56:32.526484 (+775139us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8}
I20251026 02:56:32.527029 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_34881d9847d5463297830bf6cc4ed1e6" } modify_external_catalogs: true
I20251026 02:56:32.528040 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0635cec42beb4731b5362de5a1e6b806"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.528650 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0635cec42beb4731b5362de5a1e6b806 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.529160 2375 tablet_bootstrap.cc:492] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:32.529333 2375 ts_tablet_manager.cc:1403] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.397s user 0.021s sys 0.047s
I20251026 02:56:32.530014 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet a08efd37968846d2958e54831488235c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:32.530081 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 0fe0e06a86e241bc91a1db4219fe25bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:32.530196 2087 tablet_service.cc:1552] Processing DeleteTablet for tablet 634d913c98c842a49d97a54f23752d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:32.530591 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 863ce9e1adb444b7b1399b968ea79bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:32.530560 2375 raft_consensus.cc:359] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.530745 2375 raft_consensus.cc:385] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.530843 2375 raft_consensus.cc:740] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.531091 2375 consensus_queue.cc:260] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.534734 2375 ts_tablet_manager.cc:1434] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20251026 02:56:32.534809 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31b6e08597bd450eb1875201cf594a82 consensus state has no leader
I20251026 02:56:32.535099 2375 tablet_bootstrap.cc:492] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:32.536994 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2e3bf8087ce7480584b9b66c8270c451"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.537604 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e3bf8087ce7480584b9b66c8270c451 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.537647 2375 tablet_bootstrap.cc:654] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.540535 2381 tablet_bootstrap.cc:492] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:32.540694 2381 ts_tablet_manager.cc:1403] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.000s
I20251026 02:56:32.541422 2381 raft_consensus.cc:359] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.541581 2381 raft_consensus.cc:385] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.541628 2381 raft_consensus.cc:740] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.541847 2381 consensus_queue.cc:260] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.542343 2381 ts_tablet_manager.cc:1434] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:32.544082 1659 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 548a7f9628ac4623b2c2f723f83c1e45 (table default.Table_f8baefeba2b54cb39affb2d4f1b0bef8 [id=d9a6b0b6f3e246a3aa8fbd1167205264]). Expected version 2 got 0
I20251026 02:56:32.547515 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b6138cb4ba3c4b70bcec42f1ce989668" } modify_external_catalogs: true
I20251026 02:56:32.547693 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b6138cb4ba3c4b70bcec42f1ce989668" } modify_external_catalogs: true
I20251026 02:56:32.548610 2381 tablet_bootstrap.cc:492] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:32.549638 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 903e65fa8c214dc9a50cb97cab101d22 (table default.Table_2d168c00cb884360bdb2475bd391f576 [id=65b5770c4b9243b892bfab963a439bb9]). Expected version 4 got 3
I20251026 02:56:32.552793 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f51a5406d89743649dab37ef175a4666" } modify_external_catalogs: true
I20251026 02:56:32.553437 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f51a5406d89743649dab37ef175a4666" } modify_external_catalogs: true
I20251026 02:56:32.555469 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 84ad6c72a8704c148411881760848df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.555727 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet aa399e6407fd4c4d9871d3d5ccf894ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.556864 2119 raft_consensus.cc:2468] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:32.557375 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_34881d9847d5463297830bf6cc4ed1e6" } modify_external_catalogs: true
I20251026 02:56:32.564033 2381 tablet_bootstrap.cc:654] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.571309 1659 catalog_manager.cc:5649] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:32.571904 1659 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 94886ad6d6bb45f39261725169774ac9 (table default.Table_d23ac2c9657b4adeaac9545d5bc9ae41 [id=9e897d33e6374ad4bc3c8acaae512645]). Expected version 1 got 0
I20251026 02:56:32.573666 1659 catalog_manager.cc:5649] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:32.574079 2381 tablet_bootstrap.cc:492] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:32.574216 2381 ts_tablet_manager.cc:1403] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20251026 02:56:32.574327 1659 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 049a915b35224cdaa38df77c734a0f01 (table default.Table_1818605e0bc548938337de5d5fc7a781 [id=f1f9d2ad39b34abbbe989bc3fac62b6c]). Expected version 4 got 0
I20251026 02:56:32.574898 1659 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (table default.Table_1fa437ef7f89450bb6f672a1fb491baa [id=44c37945ff9f48e8b2c07648394ac177]). Expected version 1 got 0
I20251026 02:56:32.574916 2381 raft_consensus.cc:359] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.575101 2381 raft_consensus.cc:385] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.575151 2381 raft_consensus.cc:740] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.575363 2381 consensus_queue.cc:260] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.576097 2381 ts_tablet_manager.cc:1434] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:32.576398 2381 tablet_bootstrap.cc:492] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:32.578002 2381 tablet_bootstrap.cc:654] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.577999 2375 tablet_bootstrap.cc:492] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:32.578209 2375 ts_tablet_manager.cc:1403] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20251026 02:56:32.579159 3096 tablet.cc:1722] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:32.579195 2375 raft_consensus.cc:359] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.579427 2375 raft_consensus.cc:385] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.579540 2375 raft_consensus.cc:740] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.580544 2962 raft_consensus.cc:515] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.581682 2962 leader_election.cc:290] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.581987 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa399e6407fd4c4d9871d3d5ccf894ec" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:32.582226 1974 raft_consensus.cc:3060] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.587797 2375 consensus_queue.cc:260] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.588551 2375 ts_tablet_manager.cc:1434] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20251026 02:56:32.588902 2375 tablet_bootstrap.cc:492] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:32.590791 2375 tablet_bootstrap.cc:654] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.594771 2195 leader_election.cc:304] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.595211 3065 raft_consensus.cc:2804] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:32.595320 3065 raft_consensus.cc:493] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:32.595381 3065 raft_consensus.cc:3060] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.599022 2195 leader_election.cc:304] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.599429 3028 raft_consensus.cc:2804] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:32.599582 3028 raft_consensus.cc:697] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:32.599905 3028 consensus_queue.cc:237] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.604166 3006 raft_consensus.cc:493] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.604399 3006 raft_consensus.cc:515] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.605108 3006 leader_election.cc:290] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.605937 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfc3aed64ad0439cb22fd6815c973d87" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:32.607113 2053 leader_election.cc:343] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.607292 2053 leader_election.cc:304] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.607999 2934 raft_consensus.cc:2749] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:32.608139 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e56b259a314422aba35eb129021bbeb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:32.608412 2119 raft_consensus.cc:2468] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:32.609014 2053 leader_election.cc:343] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.612419 3028 raft_consensus.cc:493] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.612615 3028 raft_consensus.cc:515] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.613314 3028 leader_election.cc:290] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.614251 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfc3aed64ad0439cb22fd6815c973d87" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:32.614351 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "863ce9e1adb444b7b1399b968ea79bb4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:32.614904 2194 leader_election.cc:343] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.614919 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47bfff40c181432896d6898ea50cd025" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:32.616387 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "863ce9e1adb444b7b1399b968ea79bb4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:32.616598 2119 raft_consensus.cc:2468] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:32.617389 2053 leader_election.cc:343] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.617527 2053 leader_election.cc:304] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.617937 3006 raft_consensus.cc:2749] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:32.618552 2195 leader_election.cc:343] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ced7ad6a601d432cbc13b3199a0506e4
I20251026 02:56:32.618709 2195 leader_election.cc:304] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.618947 2962 raft_consensus.cc:2749] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:32.620493 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 03aed9babfbe4acdad21d8ee22dbac1d (DEFAULT_TABLE table=default.Table_481575c8f05747c2954c9c79a0ebde9c [id=91d789aff2464f73b60e03b52126a396]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.623911 2378 tablet_bootstrap.cc:492] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:32.624102 2378 ts_tablet_manager.cc:1403] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.300s user 0.031s sys 0.080s
I20251026 02:56:32.625136 2378 raft_consensus.cc:359] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.625370 2378 raft_consensus.cc:385] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.625456 2378 raft_consensus.cc:740] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.625728 2378 consensus_queue.cc:260] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.628669 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7af610f8acba426b84a2c85d1d4f1c5c" } new_table_name: "default.Table_0e6ed600363d4be391d9f26eaa01e256" modify_external_catalogs: true
I20251026 02:56:32.630369 2378 ts_tablet_manager.cc:1434] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251026 02:56:32.630707 2378 tablet_bootstrap.cc:492] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:32.631835 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet 6a3491b303f94fe2a2c838297f73d464 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:32.632516 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58d60bf2a9e546e6a9474a5af6d5273a has no consensus state
I20251026 02:56:32.632794 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet a70d70abc3a34dac8cf9a747903fa78c from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:32.633455 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 882fe81da21344ed99ea735fa79f73c0 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:32.634196 2378 tablet_bootstrap.cc:654] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.636265 2375 tablet_bootstrap.cc:492] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:32.636437 2375 ts_tablet_manager.cc:1403] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.012s
I20251026 02:56:32.637252 2375 raft_consensus.cc:359] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.637429 2375 raft_consensus.cc:385] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.637513 2375 raft_consensus.cc:740] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.637660 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 86d8e1e21c5f440d8baf76c357d01ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.637727 2375 consensus_queue.cc:260] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.638339 2375 ts_tablet_manager.cc:1434] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251026 02:56:32.638629 1647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.150.60:33729 (error 108) [suppressed 7 similar messages]
I20251026 02:56:32.638799 2375 tablet_bootstrap.cc:492] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:32.641407 2375 tablet_bootstrap.cc:654] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.641376 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_679529c2b6914f4bbcba4a245a8cbb08" } modify_external_catalogs: true
I20251026 02:56:32.641636 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_679529c2b6914f4bbcba4a245a8cbb08" } modify_external_catalogs: true
I20251026 02:56:32.643031 2378 tablet_bootstrap.cc:492] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:32.643190 2378 ts_tablet_manager.cc:1403] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20251026 02:56:32.643895 2378 raft_consensus.cc:359] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.644071 2378 raft_consensus.cc:385] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.644155 2378 raft_consensus.cc:740] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.644713 2378 consensus_queue.cc:260] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.647814 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2e3225d53ae74a61a8f6e6c91d55f8e1" } new_table_name: "default.Table_8db1d3c49d8d44ffa6c3160fa1720e20" modify_external_catalogs: true
I20251026 02:56:32.649086 2378 ts_tablet_manager.cc:1434] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251026 02:56:32.649590 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_aca29902287d4d8fa34e85821bfa93db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.650120 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aca29902287d4d8fa34e85821bfa93db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:32.652232 3130 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 02:56:32.637355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.150.60:33729 (local address 127.1.150.2:41827)
1026 02:56:32.639312 (+ 1957us) negotiation.cc:107] Waiting for socket to connect
1026 02:56:32.639333 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1026 02:56:32.640399 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 02:56:32.651761 (+ 11362us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1820,"thread_start_us":3629,"threads_started":1}
I20251026 02:56:32.652666 1974 raft_consensus.cc:2468] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:32.652982 3030 raft_consensus.cc:493] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:32.653167 3134 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 02:56:32.646012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.150.60:33729 (local address 127.0.0.1:34770)
1026 02:56:32.646497 (+ 485us) negotiation.cc:107] Waiting for socket to connect
1026 02:56:32.646516 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1026 02:56:32.646632 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 02:56:32.652849 (+ 6217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":251,"threads_started":1}
I20251026 02:56:32.653165 2194 leader_election.cc:304] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:32.653146 3030 raft_consensus.cc:515] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.653874 3030 leader_election.cc:290] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.654520 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22e4a433801a4cb58062d3e76c760b69" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:32.655589 2378 tablet_bootstrap.cc:492] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:32.657682 2378 tablet_bootstrap.cc:654] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.657656 3065 raft_consensus.cc:515] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.658248 3065 leader_election.cc:290] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.658411 3065 raft_consensus.cc:2804] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:32.658504 3065 raft_consensus.cc:697] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:32.658731 3065 consensus_queue.cc:237] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.660421 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6d6f024d8b943d5acf69b64bc7c7214" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:32.661052 2194 leader_election.cc:343] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.664713 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03aed9babfbe4acdad21d8ee22dbac1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.666951 2375 tablet_bootstrap.cc:492] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:32.667169 2375 ts_tablet_manager.cc:1403] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.004s
I20251026 02:56:32.668150 2375 raft_consensus.cc:359] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.668434 2375 raft_consensus.cc:385] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.668519 2375 raft_consensus.cc:740] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.668859 2375 consensus_queue.cc:260] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.669591 2375 ts_tablet_manager.cc:1434] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251026 02:56:32.670012 2375 tablet_bootstrap.cc:492] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:32.670334 1647 consensus_peers.cc:597] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 -> Peer 751b719b75ce4bc2a88a19091230ecb7 (127.1.150.60:33729): Couldn't send request to peer 751b719b75ce4bc2a88a19091230ecb7. Status: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:32.671044 3131 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 02:56:32.638629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.150.60:33729 (local address 127.1.150.1:39771)
1026 02:56:32.670517 (+ 31888us) negotiation.cc:107] Waiting for socket to connect
1026 02:56:32.670693 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":31749,"threads_started":1}
I20251026 02:56:32.672027 2375 tablet_bootstrap.cc:654] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.673801 2378 tablet_bootstrap.cc:492] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:32.674151 2378 ts_tablet_manager.cc:1403] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.002s
I20251026 02:56:32.675136 2378 raft_consensus.cc:359] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.675395 2378 raft_consensus.cc:385] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.675510 2378 raft_consensus.cc:740] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.675841 2378 consensus_queue.cc:260] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.680843 2378 ts_tablet_manager.cc:1434] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251026 02:56:32.685326 2378 tablet_bootstrap.cc:492] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:32.687521 2378 tablet_bootstrap.cc:654] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.703667 3081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:32.703917 3081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: This master's current role is: LEADER
I20251026 02:56:32.709511 3006 raft_consensus.cc:493] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.709805 3006 raft_consensus.cc:515] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.710669 3006 leader_election.cc:290] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.739796 1946 tablet_service.cc:1505] Processing CreateTablet for tablet d7313a6396c541d7ac6d8ca2e219af90 (DEFAULT_TABLE table=default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.744372 1961 tablet_service.cc:1505] Processing CreateTablet for tablet c13d89fc829049428363ede24c350ee5 (DEFAULT_TABLE table=default.Table_c8a0cf8b6d464c81825ff3b2d17afeaf [id=431f5af9a9574a47889129ededaeed3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.748991 2091 tablet_service.cc:1505] Processing CreateTablet for tablet c13d89fc829049428363ede24c350ee5 (DEFAULT_TABLE table=default.Table_c8a0cf8b6d464c81825ff3b2d17afeaf [id=431f5af9a9574a47889129ededaeed3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.749257 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 5b562b7b51f94b268c667650df199fbd (DEFAULT_TABLE table=default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.749615 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13d89fc829049428363ede24c350ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.756539 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 03aed9babfbe4acdad21d8ee22dbac1d (DEFAULT_TABLE table=default.Table_481575c8f05747c2954c9c79a0ebde9c [id=91d789aff2464f73b60e03b52126a396]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.757310 3006 raft_consensus.cc:493] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.757516 3006 raft_consensus.cc:515] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.758433 3006 leader_election.cc:290] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.758550 2086 tablet_service.cc:1505] Processing CreateTablet for tablet d7313a6396c541d7ac6d8ca2e219af90 (DEFAULT_TABLE table=default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.759909 2377 ts_tablet_manager.cc:1929] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:32.762821 2376 ts_tablet_manager.cc:1929] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:32.763365 2376 log.cc:1199] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/7845ebd3e9ca4897b8fb19a48a010315
I20251026 02:56:32.763909 2376 ts_tablet_manager.cc:1950] T 7845ebd3e9ca4897b8fb19a48a010315 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:32.764539 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7313a6396c541d7ac6d8ca2e219af90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.764784 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b562b7b51f94b268c667650df199fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.767045 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13d89fc829049428363ede24c350ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.770943 3006 raft_consensus.cc:493] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.771148 3006 raft_consensus.cc:515] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.771899 3006 leader_election.cc:290] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.777958 2381 tablet_bootstrap.cc:492] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:32.778153 2381 ts_tablet_manager.cc:1403] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.202s user 0.027s sys 0.045s
I20251026 02:56:32.779117 2381 raft_consensus.cc:359] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.779341 2381 raft_consensus.cc:385] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.779426 2381 raft_consensus.cc:740] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.779655 2381 consensus_queue.cc:260] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:32.783874 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 165310us
I20251026 02:56:32.784180 2195 leader_election.cc:304] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:32.784570 2195 leader_election.cc:343] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: aa399e6407fd4c4d9871d3d5ccf894ec
I20251026 02:56:32.784602 2962 raft_consensus.cc:2804] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:32.784778 2962 raft_consensus.cc:493] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:32.784857 2962 raft_consensus.cc:3060] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.760131 2377 log.cc:1199] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/122de8616b8f4fcf88971d5738c47e82
I20251026 02:56:32.785420 2377 ts_tablet_manager.cc:1950] T 122de8616b8f4fcf88971d5738c47e82 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:32.786175 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 407) took 4397 ms. Trace:
I20251026 02:56:32.786301 2377 rpcz_store.cc:276] 1026 02:56:28.388910 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.388967 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:32.301118 (+3912151us) ts_tablet_manager.cc:1151] Deleting tablet 122de8616b8f4fcf88971d5738c47e82
1026 02:56:32.301126 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.496264 (+195138us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.759699 (+263435us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.786153 (+ 26454us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3895042}
I20251026 02:56:32.786800 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 411) took 4368 ms. Trace:
I20251026 02:56:32.786926 2377 rpcz_store.cc:276] 1026 02:56:28.418644 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.418698 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:32.786682 (+4367984us) ts_tablet_manager.cc:1151] Deleting tablet a53f25688e3b45dca68984e2c543d094
1026 02:56:32.786694 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.786785 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4367785}
I20251026 02:56:32.787297 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 410) took 4371 ms. Trace:
I20251026 02:56:32.787400 2377 rpcz_store.cc:276] 1026 02:56:28.415878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.415928 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:32.787210 (+4371282us) ts_tablet_manager.cc:1151] Deleting tablet 17d2cc6b16d34331bb80c576c0cd2ce0
1026 02:56:32.787219 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.787285 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4361152}
I20251026 02:56:32.787708 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 412) took 4368 ms. Trace:
I20251026 02:56:32.787809 2377 rpcz_store.cc:276] 1026 02:56:28.419069 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.419137 (+ 68us) service_pool.cc:225] Handling call
1026 02:56:32.787630 (+4368493us) ts_tablet_manager.cc:1151] Deleting tablet 209fa71af13142a58efb9f3dab3eecd3
1026 02:56:32.787637 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.787697 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4359271}
I20251026 02:56:32.788144 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 413) took 4368 ms. Trace:
I20251026 02:56:32.788269 2377 rpcz_store.cc:276] 1026 02:56:28.419283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.419333 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:32.788059 (+4368726us) ts_tablet_manager.cc:1151] Deleting tablet aa10cd990a944eab8d9eb66aaf20080c
1026 02:56:32.788067 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.788132 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4359283}
I20251026 02:56:32.789080 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 414) took 4148 ms. Trace:
I20251026 02:56:32.803465 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 5b562b7b51f94b268c667650df199fbd (DEFAULT_TABLE table=default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.805496 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e75c4f87d9d54b479fba42c803c3bc9b" } modify_external_catalogs: true
I20251026 02:56:32.805719 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e75c4f87d9d54b479fba42c803c3bc9b" } modify_external_catalogs: true
I20251026 02:56:32.806632 2372 ts_tablet_manager.cc:1929] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:32.806770 2372 log.cc:1199] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/0efdcc36ff5b47379e0a76512e05d379
I20251026 02:56:32.807209 2372 ts_tablet_manager.cc:1950] T 0efdcc36ff5b47379e0a76512e05d379 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:32.784942 3028 consensus_queue.cc:1048] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:32.809129 2377 rpcz_store.cc:276] 1026 02:56:28.640730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.640820 (+ 90us) service_pool.cc:225] Handling call
1026 02:56:32.788958 (+4148138us) ts_tablet_manager.cc:1151] Deleting tablet efd751ee4b1d433198dcd00c081f648d
1026 02:56:32.788970 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.789064 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4147276}
I20251026 02:56:32.809469 2377 tablet_replica.cc:333] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:32.809659 2377 raft_consensus.cc:2243] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:32.809798 2377 raft_consensus.cc:2272] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:32.809870 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b562b7b51f94b268c667650df199fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.811061 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03aed9babfbe4acdad21d8ee22dbac1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.811550 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7313a6396c541d7ac6d8ca2e219af90. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:32.784967 2195 leader_election.cc:343] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: aa399e6407fd4c4d9871d3d5ccf894ec
I20251026 02:56:32.814306 2195 leader_election.cc:304] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.814877 3028 raft_consensus.cc:2804] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:32.814960 3028 raft_consensus.cc:493] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:32.815047 3028 raft_consensus.cc:3060] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.815659 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 122de8616b8f4fcf88971d5738c47e82 (table default.Table_719b11bde46041c5a20750db14381807 [id=4912ce6ea2304b5a9c21dd91a7cbd56f]) successfully deleted
W20251026 02:56:32.816128 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet a53f25688e3b45dca68984e2c543d094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a53f25688e3b45dca68984e2c543d094 already in progress: creating tablet
W20251026 02:56:32.816570 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 17d2cc6b16d34331bb80c576c0cd2ce0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17d2cc6b16d34331bb80c576c0cd2ce0 already in progress: creating tablet
I20251026 02:56:32.816866 3006 raft_consensus.cc:493] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.817015 3006 raft_consensus.cc:515] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.817209 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6d6f024d8b943d5acf69b64bc7c7214" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:32.817399 2114 raft_consensus.cc:3060] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.817672 3006 leader_election.cc:290] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.807474 2381 ts_tablet_manager.cc:1434] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20251026 02:56:32.821795 2381 tablet_bootstrap.cc:492] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:32.823568 2381 tablet_bootstrap.cc:654] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.826406 1656 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet de9124fe1c8945d9b9565cb732bef20e
I20251026 02:56:32.826885 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 466) took 1872 ms. Trace:
I20251026 02:56:32.827684 2375 tablet_bootstrap.cc:492] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:32.827827 2375 ts_tablet_manager.cc:1403] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.158s user 0.007s sys 0.036s
I20251026 02:56:32.828222 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0de06ea2b0544402a979d9ddc487860f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:32.828563 2377 ts_tablet_manager.cc:1916] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:32.828681 2375 raft_consensus.cc:359] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.828853 2375 raft_consensus.cc:385] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.828930 2375 raft_consensus.cc:740] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.829128 2375 consensus_queue.cc:260] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.829734 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7da859119bd42ea9fa1290b3d4bf88c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:32.830054 2962 raft_consensus.cc:515] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.833601 3028 raft_consensus.cc:515] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.834255 3028 leader_election.cc:290] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.834867 2962 leader_election.cc:290] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.835409 2114 raft_consensus.cc:2468] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:32.835762 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 209fa71af13142a58efb9f3dab3eecd3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 209fa71af13142a58efb9f3dab3eecd3 already in progress: creating tablet
W20251026 02:56:32.838747 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet aa10cd990a944eab8d9eb66aaf20080c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa10cd990a944eab8d9eb66aaf20080c already in progress: creating tablet
W20251026 02:56:32.839301 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet efd751ee4b1d433198dcd00c081f648d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efd751ee4b1d433198dcd00c081f648d already in progress: creating tablet
I20251026 02:56:32.840189 3144 raft_consensus.cc:493] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.840341 3144 raft_consensus.cc:515] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.840893 3144 leader_election.cc:290] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.841122 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0de06ea2b0544402a979d9ddc487860f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:32.841084 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2524ee0c964345459c3e62f1e39b8f9e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:32.841289 2262 raft_consensus.cc:2468] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:32.842015 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e9b0e1d3f3f4fdbb8c58f26a4308771" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:32.843068 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 112181b9207e448f959f8311ad95a02b (DEFAULT_TABLE table=default.Table_e4a6aa2d2b9f4653b73ae643cfe689cc [id=a35cd7c911524520903e9bd07cc84224]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.843334 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22e4a433801a4cb58062d3e76c760b69" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:32.843462 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7da859119bd42ea9fa1290b3d4bf88c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:32.843981 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 03aed9babfbe4acdad21d8ee22dbac1d (DEFAULT_TABLE table=default.Table_481575c8f05747c2954c9c79a0ebde9c [id=91d789aff2464f73b60e03b52126a396]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.844442 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03aed9babfbe4acdad21d8ee22dbac1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.844895 3144 consensus_queue.cc:1048] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:32.844985 2195 leader_election.cc:304] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.845363 3144 raft_consensus.cc:2804] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:32.845480 3144 raft_consensus.cc:697] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:32.845691 3144 consensus_queue.cc:237] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.846406 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 112181b9207e448f959f8311ad95a02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.848510 2229 tablet_service.cc:1505] Processing CreateTablet for tablet d7313a6396c541d7ac6d8ca2e219af90 (DEFAULT_TABLE table=default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.848874 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 491) took 1723 ms. Trace:
I20251026 02:56:32.848917 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7313a6396c541d7ac6d8ca2e219af90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.849144 2372 rpcz_store.cc:276] 1026 02:56:31.125005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.125863 (+ 858us) service_pool.cc:225] Handling call
1026 02:56:32.310808 (+1184945us) ts_tablet_manager.cc:1151] Deleting tablet 0efdcc36ff5b47379e0a76512e05d379
1026 02:56:32.310825 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.652012 (+341187us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.806465 (+154453us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.848836 (+ 42371us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1184327}
I20251026 02:56:32.849738 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 0efdcc36ff5b47379e0a76512e05d379 (table default.Table_1972d81a4c494b7c97894d4991fed574 [id=d9f8b0fbb3ed416fb380a5717fb29dba]) successfully deleted
I20251026 02:56:32.850317 2372 tablet_replica.cc:333] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:32.851161 2053 leader_election.cc:343] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.851734 2053 leader_election.cc:304] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:32.852519 2226 tablet_service.cc:1505] Processing CreateTablet for tablet c13d89fc829049428363ede24c350ee5 (DEFAULT_TABLE table=default.Table_c8a0cf8b6d464c81825ff3b2d17afeaf [id=431f5af9a9574a47889129ededaeed3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.852495 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 112181b9207e448f959f8311ad95a02b (DEFAULT_TABLE table=default.Table_e4a6aa2d2b9f4653b73ae643cfe689cc [id=a35cd7c911524520903e9bd07cc84224]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.852869 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13d89fc829049428363ede24c350ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.853381 2375 ts_tablet_manager.cc:1434] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20251026 02:56:32.853792 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2524ee0c964345459c3e62f1e39b8f9e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:32.853593 3007 raft_consensus.cc:2804] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:32.854559 3007 raft_consensus.cc:493] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:32.854677 3007 raft_consensus.cc:3060] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.853449 2376 rpcz_store.cc:276] 1026 02:56:30.910267 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.910402 (+ 135us) service_pool.cc:225] Handling call
1026 02:56:32.504066 (+1593664us) ts_tablet_manager.cc:1151] Deleting tablet 7845ebd3e9ca4897b8fb19a48a010315
1026 02:56:32.504076 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.656923 (+152847us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.762655 (+105732us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.782426 (+ 19771us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1593507}
I20251026 02:56:32.856552 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 5b562b7b51f94b268c667650df199fbd (DEFAULT_TABLE table=default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.856904 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b562b7b51f94b268c667650df199fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.857230 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 112181b9207e448f959f8311ad95a02b (DEFAULT_TABLE table=default.Table_e4a6aa2d2b9f4653b73ae643cfe689cc [id=a35cd7c911524520903e9bd07cc84224]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.857517 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 112181b9207e448f959f8311ad95a02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.857546 2375 tablet_bootstrap.cc:492] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:32.854188 2053 leader_election.cc:343] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:32.858247 2053 leader_election.cc:343] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.860731 1915 leader_election.cc:304] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:32.861236 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "863ce9e1adb444b7b1399b968ea79bb4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:32.861366 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a5b68772a66402bbb0b7e00bf96fcc0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:32.861402 2118 raft_consensus.cc:3060] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.861490 2117 raft_consensus.cc:2468] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:32.861784 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e56b259a314422aba35eb129021bbeb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:32.861905 2119 raft_consensus.cc:3060] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.863382 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 112181b9207e448f959f8311ad95a02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.865640 2195 leader_election.cc:304] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:32.866029 3028 raft_consensus.cc:2804] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:32.866110 3028 raft_consensus.cc:493] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:32.866098 1978 raft_consensus.cc:3060] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.866173 3028 raft_consensus.cc:3060] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.866143 3030 raft_consensus.cc:2749] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:32.866128 2376 tablet_replica.cc:333] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
W20251026 02:56:32.869942 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 44406.5us
I20251026 02:56:32.872641 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 7845ebd3e9ca4897b8fb19a48a010315 (table default.Table_9c97bccbf4cb4d15a66dbb8130acb6d4 [id=10d722da66cd4661a1bb1ab82997dca8]) successfully deleted
W20251026 02:56:32.876071 2194 consensus_peers.cc:597] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:32.876269 2194 consensus_peers.cc:597] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:32.876961 2376 raft_consensus.cc:2243] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:32.877187 1915 leader_election.cc:304] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:32.877460 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e9b0e1d3f3f4fdbb8c58f26a4308771" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:32.878098 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 8306f2ece961480d906df88f40b9740b (DEFAULT_TABLE table=default.Table_2b6ab2f608284fb1aec83b83ab7da74f [id=8267517ec80f4d68a2bd7bfee11b364d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.878441 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a5b68772a66402bbb0b7e00bf96fcc0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:32.878787 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e56b259a314422aba35eb129021bbeb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:32.878530 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8306f2ece961480d906df88f40b9740b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:32.879750 2194 leader_election.cc:343] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:32.880273 2194 leader_election.cc:343] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:32.880869 2053 leader_election.cc:343] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.881120 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7a4c0546cb2e4efdb10c4f630a115607"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.881245 2053 leader_election.cc:343] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.881577 2376 raft_consensus.cc:2272] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:32.881500 2934 raft_consensus.cc:493] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:32.881734 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a4c0546cb2e4efdb10c4f630a115607 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.881765 2934 raft_consensus.cc:515] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.882157 2376 ts_tablet_manager.cc:1916] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:32.882319 2934 leader_election.cc:290] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.882570 2053 leader_election.cc:304] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.882882 2934 raft_consensus.cc:2749] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:32.883775 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 262f60fdb274476abf3533b4f47688b8
W20251026 02:56:32.883435 2053 leader_election.cc:343] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.884891 2053 leader_election.cc:304] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.886462 2934 raft_consensus.cc:2749] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:32.886771 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet dbfe80d388644d0488cb2431a334c908 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.887769 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fa3b58cf82f242cebd20cc1ea9b2133d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.888366 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa3b58cf82f242cebd20cc1ea9b2133d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.888600 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "863ce9e1adb444b7b1399b968ea79bb4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:32.891144 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 262f60fdb274476abf3533b4f47688b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:32.892270 2372 raft_consensus.cc:2243] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:32.892634 2372 pending_rounds.cc:70] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48: Trying to abort 1 pending ops.
I20251026 02:56:32.893121 2372 pending_rounds.cc:77] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888515291959296 op_type: NO_OP noop_request { }
I20251026 02:56:32.893445 2372 raft_consensus.cc:2889] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:32.893898 2372 raft_consensus.cc:2272] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:32.895565 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 262f60fdb274476abf3533b4f47688b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.896402 2372 ts_tablet_manager.cc:1916] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:32.898131 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3b77bb6b4722449c9699b548a9ef6314
I20251026 02:56:32.899227 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b77bb6b4722449c9699b548a9ef6314 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:32.899699 2195 consensus_peers.cc:597] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:32.896502 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 8306f2ece961480d906df88f40b9740b (DEFAULT_TABLE table=default.Table_2b6ab2f608284fb1aec83b83ab7da74f [id=8267517ec80f4d68a2bd7bfee11b364d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:32.900451 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8306f2ece961480d906df88f40b9740b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.901585 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet b1a2522801364693be0bda6fa45a697e with delete_type TABLET_DATA_DELETED (replaced by tablet ff242ef9541a42d793c4484190956da6) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.901825 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b77bb6b4722449c9699b548a9ef6314 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:32.902010 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b77bb6b4722449c9699b548a9ef6314 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.908351 2962 raft_consensus.cc:493] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.908519 2962 raft_consensus.cc:515] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:32.909257 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4d64902618d46428ff345bc190b03bd consensus state has no leader
I20251026 02:56:32.909451 2962 leader_election.cc:290] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.910204 1657 catalog_manager.cc:5649] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:32.911275 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 548a7f9628ac4623b2c2f723f83c1e45 (table default.Table_f8baefeba2b54cb39affb2d4f1b0bef8 [id=d9a6b0b6f3e246a3aa8fbd1167205264]). Expected version 2 got 0
I20251026 02:56:32.912969 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 262f60fdb274476abf3533b4f47688b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:32.913525 2226 tablet_service.cc:1505] Processing CreateTablet for tablet 8306f2ece961480d906df88f40b9740b (DEFAULT_TABLE table=default.Table_2b6ab2f608284fb1aec83b83ab7da74f [id=8267517ec80f4d68a2bd7bfee11b364d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:32.914093 2194 leader_election.cc:343] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.914345 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f439acecb4bb4404a14e11f5824c654b" } new_table_name: "default.Table_6b9199ff4e884b4891446da6e6e6195d" modify_external_catalogs: true
W20251026 02:56:32.916453 2195 leader_election.cc:343] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: a70d70abc3a34dac8cf9a747903fa78c
I20251026 02:56:32.917033 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a70d70abc3a34dac8cf9a747903fa78c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:32.917428 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c1abfb93d24a4f2a97d8d87a06c34ea5" } new_table_name: "default.Table_1571e787564342cd8a0f26f710f89b0c" modify_external_catalogs: true
I20251026 02:56:32.919656 1662 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 47c214fd4cc249cd96b5e908e078033a
W20251026 02:56:32.921141 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 046a34f9408b406ba08471421f422064 consensus state has no leader
W20251026 02:56:32.922066 2194 leader_election.cc:343] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.922219 2194 leader_election.cc:304] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.922484 3150 raft_consensus.cc:2749] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:32.922413 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_bc0096c9da9c4603a0d2ea3538682695"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.916903 2053 leader_election.cc:343] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.923008 2053 leader_election.cc:304] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:32.923071 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc0096c9da9c4603a0d2ea3538682695 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.924515 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_61f1d10f16b1467aaab14fe4fbaef1bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:32.925017 2934 raft_consensus.cc:2749] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:32.925139 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61f1d10f16b1467aaab14fe4fbaef1bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.908639 2375 tablet_bootstrap.cc:654] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.926306 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "276bd3b97c97426daf802e1f57feb265" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:32.929004 3150 raft_consensus.cc:493] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:32.929718 3154 raft_consensus.cc:493] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.930119 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "276bd3b97c97426daf802e1f57feb265" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:32.930045 3150 raft_consensus.cc:515] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:32.930507 2053 leader_election.cc:343] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.930860 3150 leader_election.cc:290] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
W20251026 02:56:32.931897 2195 leader_election.cc:343] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9a08e8b7eb1949408f2151f2d7257e9d
W20251026 02:56:32.933091 2194 leader_election.cc:343] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 9a08e8b7eb1949408f2151f2d7257e9d
I20251026 02:56:32.933259 2194 leader_election.cc:304] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.934844 3028 raft_consensus.cc:515] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.935462 3028 leader_election.cc:290] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:32.935657 3028 raft_consensus.cc:2749] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:32.936677 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8306f2ece961480d906df88f40b9740b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:32.938417 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a5b68772a66402bbb0b7e00bf96fcc0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:32.938678 2114 raft_consensus.cc:3060] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.939476 3007 raft_consensus.cc:515] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.940207 3007 leader_election.cc:290] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:32.941769 2053 leader_election.cc:343] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.941870 2053 leader_election.cc:304] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:32.944331 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0de06ea2b0544402a979d9ddc487860f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:32.944494 2263 raft_consensus.cc:3060] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:32.945147 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d48a43604ff140f0b7103cc9fb517a5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:32.945782 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d48a43604ff140f0b7103cc9fb517a5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.951535 2934 raft_consensus.cc:2749] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:32.953073 2934 consensus_queue.cc:1048] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:32.955467 2381 tablet_bootstrap.cc:492] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:32.955614 2381 ts_tablet_manager.cc:1403] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.134s user 0.003s sys 0.000s
I20251026 02:56:32.956432 2381 raft_consensus.cc:359] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.956595 2381 raft_consensus.cc:385] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.956643 2381 raft_consensus.cc:740] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.956888 2381 consensus_queue.cc:260] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.957513 2381 ts_tablet_manager.cc:1434] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:32.957849 2381 tablet_bootstrap.cc:492] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:32.959970 2381 tablet_bootstrap.cc:654] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:32.964977 2375 tablet_bootstrap.cc:492] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:32.965126 2375 ts_tablet_manager.cc:1403] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.108s user 0.003s sys 0.000s
I20251026 02:56:32.965216 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 6ea364c4a5cf4f3f99d9ff6904add697
I20251026 02:56:32.965935 2375 raft_consensus.cc:359] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.966122 2375 raft_consensus.cc:385] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.966182 2375 raft_consensus.cc:740] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:32.966406 2375 consensus_queue.cc:260] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:32.966725 2381 tablet_bootstrap.cc:492] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:32.966710 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ea364c4a5cf4f3f99d9ff6904add697 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:32.939559 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a5b68772a66402bbb0b7e00bf96fcc0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:32.966858 2381 ts_tablet_manager.cc:1403] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20251026 02:56:32.940657 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0de06ea2b0544402a979d9ddc487860f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:32.944819 2376 ts_tablet_manager.cc:1929] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:32.967344 2376 log.cc:1199] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/8dbc12d49b094dbeadad2e76916ba9af
I20251026 02:56:32.967463 3154 raft_consensus.cc:515] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.967814 2376 ts_tablet_manager.cc:1950] T 8dbc12d49b094dbeadad2e76916ba9af P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:32.962158 2378 tablet_bootstrap.cc:492] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:32.968349 2378 ts_tablet_manager.cc:1403] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.283s user 0.061s sys 0.056s
W20251026 02:56:32.969103 2194 leader_election.cc:343] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:32.969121 1915 leader_election.cc:343] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: f8533f1c191042ee9f2c0b99972fe931
W20251026 02:56:32.969283 1916 leader_election.cc:343] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: f8533f1c191042ee9f2c0b99972fe931
W20251026 02:56:32.969383 2053 leader_election.cc:343] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:32.969401 1916 leader_election.cc:304] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:32.969399 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ea364c4a5cf4f3f99d9ff6904add697 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:32.970141 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c4160ae9bbe94aa8a45dbf23888f7aaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:32.970217 2378 raft_consensus.cc:359] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.970374 2378 raft_consensus.cc:385] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:32.970459 2378 raft_consensus.cc:740] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
W20251026 02:56:32.970767 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4160ae9bbe94aa8a45dbf23888f7aaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:32.970695 2378 consensus_queue.cc:260] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.973698 3150 consensus_queue.cc:1048] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:32.975210 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ea364c4a5cf4f3f99d9ff6904add697 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:32.977093 3154 leader_election.cc:290] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:32.978163 3154 raft_consensus.cc:2749] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:32.980741 3154 raft_consensus.cc:493] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:32.983837 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_dafd0323dc7542da8dca42d770f1455f" } new_table_name: "default.Table_9b0942af08414e7bb088a14c781f0edd" modify_external_catalogs: true
I20251026 02:56:32.981951 3096 tablet.cc:1722] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:32.985514 3154 raft_consensus.cc:515] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:32.987494 3030 raft_consensus.cc:2749] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:32.992946 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84ad6c72a8704c148411881760848df4" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:32.993083 3154 leader_election.cc:290] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:32.993577 1915 leader_election.cc:343] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: Tablet not RUNNING: STOPPED
I20251026 02:56:32.994292 2375 ts_tablet_manager.cc:1434] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20251026 02:56:33.001133 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 465) took 2090 ms. Trace:
I20251026 02:56:33.001302 2376 rpcz_store.cc:276] 1026 02:56:30.910144 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.910640 (+ 496us) service_pool.cc:225] Handling call
1026 02:56:32.855031 (+1944391us) ts_tablet_manager.cc:1151] Deleting tablet 8dbc12d49b094dbeadad2e76916ba9af
1026 02:56:32.860451 (+ 5420us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.934327 (+ 73876us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:32.944679 (+ 10352us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.001092 (+ 56413us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1944189}
I20251026 02:56:33.001765 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 467) took 2053 ms. Trace:
I20251026 02:56:33.001871 2376 rpcz_store.cc:276] 1026 02:56:30.948748 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.948813 (+ 65us) service_pool.cc:225] Handling call
1026 02:56:33.001620 (+2052807us) ts_tablet_manager.cc:1151] Deleting tablet 8fe03778a1e842829cab25c90597a3cd
1026 02:56:33.001629 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.001731 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2052548}
I20251026 02:56:33.002161 2376 tablet_replica.cc:333] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:33.002354 2376 raft_consensus.cc:2243] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:33.002569 2376 raft_consensus.cc:2272] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:33.003196 2376 ts_tablet_manager.cc:1916] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:33.007263 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5b564a01de564f7aa92845f3d962af11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.007853 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b564a01de564f7aa92845f3d962af11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.009397 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 8dbc12d49b094dbeadad2e76916ba9af (table default.Table_cdb10d198a394aac9033a98152fd2d94 [id=1d0784496733469fbff42783f90d56d2]) successfully deleted
W20251026 02:56:33.009680 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 8fe03778a1e842829cab25c90597a3cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fe03778a1e842829cab25c90597a3cd already in progress: creating tablet
I20251026 02:56:33.014104 3150 consensus_queue.cc:1048] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:33.027462 1659 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_71583d73a49142ae9fec31193fb53542 [id=e86d9fc892a8409590c555e25b27a12f] alter complete (version 1)
I20251026 02:56:33.027547 3006 raft_consensus.cc:493] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.027683 3006 raft_consensus.cc:515] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.028371 3006 leader_election.cc:290] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.029328 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7062c5ff91e14369ab69a2d199585c55" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.029595 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7062c5ff91e14369ab69a2d199585c55" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:33.029848 2053 leader_election.cc:343] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.030602 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet f6d6f024d8b943d5acf69b64bc7c7214 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a3491b303f94fe2a2c838297f73d464) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.030831 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 5caeb11322fe48348ebb523967152187 with delete_type TABLET_DATA_DELETED (replaced by tablet d43b6f22dd4f4f9c9657f21514550312) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.031246 2118 raft_consensus.cc:2468] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:33.031646 2263 raft_consensus.cc:2468] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:33.033663 2053 leader_election.cc:304] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.034047 3006 raft_consensus.cc:2804] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:33.034152 3006 raft_consensus.cc:697] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:33.034400 3006 consensus_queue.cc:237] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.034564 2377 ts_tablet_manager.cc:1929] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:33.034657 2377 log.cc:1199] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/84ad6c72a8704c148411881760848df4
I20251026 02:56:33.034930 2377 ts_tablet_manager.cc:1950] T 84ad6c72a8704c148411881760848df4 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:33.036616 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 2524ee0c964345459c3e62f1e39b8f9e with delete_type TABLET_DATA_DELETED (replaced by tablet 5a00bc04b3654f90af342986751344e6) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.036852 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 0de06ea2b0544402a979d9ddc487860f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.037271 2195 leader_election.cc:304] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.038813 1659 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 5226d4516ce24ecda51dab35c86da132 (table default.Table_a6b94cbe15a34811a6a613f0d749a414 [id=c8435cef755d4de69c33f599df1c0f75]). Expected version 1 got 0
I20251026 02:56:33.040915 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_51306871ca71474db5412b80acf6f5e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.041605 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51306871ca71474db5412b80acf6f5e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.041688 2375 tablet_bootstrap.cc:492] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:33.041755 2053 leader_election.cc:343] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.041882 2053 leader_election.cc:304] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.042122 2934 raft_consensus.cc:2749] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:33.042806 3144 raft_consensus.cc:2804] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:33.042949 3144 raft_consensus.cc:697] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:33.043196 2375 tablet_bootstrap.cc:654] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.043169 3144 consensus_queue.cc:237] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:33.046599 2194 consensus_peers.cc:597] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:33.056298 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 0de06ea2b0544402a979d9ddc487860f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.056887 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e56b259a314422aba35eb129021bbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.057322 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet f6d6f024d8b943d5acf69b64bc7c7214 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a3491b303f94fe2a2c838297f73d464) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.057726 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 122de8616b8f4fcf88971d5738c47e82 with delete_type TABLET_DATA_DELETED (replaced by tablet 81775055d98349f696cdab6171ffd58b) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.058107 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 863ce9e1adb444b7b1399b968ea79bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.061730 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet ced7ad6a601d432cbc13b3199a0506e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.061271 2375 tablet_bootstrap.cc:492] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.062523 2375 ts_tablet_manager.cc:1403] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251026 02:56:33.063459 2375 raft_consensus.cc:359] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.063655 2375 raft_consensus.cc:385] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.063741 2375 raft_consensus.cc:740] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.064431 2381 raft_consensus.cc:359] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.064749 2381 raft_consensus.cc:385] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.064864 2381 raft_consensus.cc:740] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.065423 2381 consensus_queue.cc:260] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.065819 2375 consensus_queue.cc:260] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.066989 2381 ts_tablet_manager.cc:1434] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.100s user 0.001s sys 0.001s
I20251026 02:56:33.067600 2375 ts_tablet_manager.cc:1434] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251026 02:56:33.068028 2381 tablet_bootstrap.cc:492] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:33.068482 2375 tablet_bootstrap.cc:492] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.069696 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_890ffb18fe5a49f1938a0fca64189907"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.070518 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_890ffb18fe5a49f1938a0fca64189907 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.071731 2381 tablet_bootstrap.cc:654] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.073526 2375 tablet_bootstrap.cc:654] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:33.077222 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34c6a7c774044411a4733b115080c4c6 consensus state has no leader
I20251026 02:56:33.077606 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 903e65fa8c214dc9a50cb97cab101d22 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:33.098578 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8fe7825fa8d46669772cf84bc4b3afc consensus state has no leader
W20251026 02:56:33.096503 2053 consensus_peers.cc:597] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:33.082933 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet f85c2745a371455a8dda67f31a19c9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.107554 2378 ts_tablet_manager.cc:1434] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.139s user 0.004s sys 0.001s
I20251026 02:56:33.108089 2378 tablet_bootstrap.cc:492] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:33.111151 1916 leader_election.cc:343] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 84ad6c72a8704c148411881760848df4
I20251026 02:56:33.111181 2378 tablet_bootstrap.cc:654] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.111440 1916 leader_election.cc:304] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:33.112538 3164 raft_consensus.cc:2749] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:33.113405 2381 tablet_bootstrap.cc:492] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:33.113757 2375 tablet_bootstrap.cc:492] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.113967 2375 ts_tablet_manager.cc:1403] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.001s
I20251026 02:56:33.114764 2375 raft_consensus.cc:359] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.115082 2381 ts_tablet_manager.cc:1403] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.002s
I20251026 02:56:33.119019 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 424) took 4415 ms. Trace:
I20251026 02:56:33.119415 2375 raft_consensus.cc:385] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.119496 2375 raft_consensus.cc:740] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.119679 2375 consensus_queue.cc:260] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.120149 2375 ts_tablet_manager.cc:1434] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251026 02:56:33.120427 2375 tablet_bootstrap.cc:492] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.120352 2377 rpcz_store.cc:276] 1026 02:56:28.703774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.704272 (+ 498us) service_pool.cc:225] Handling call
1026 02:56:32.809420 (+4105148us) ts_tablet_manager.cc:1151] Deleting tablet 84ad6c72a8704c148411881760848df4
1026 02:56:32.809431 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.943230 (+133799us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.034447 (+ 91217us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.118996 (+ 84549us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4093960}
I20251026 02:56:33.119230 2381 raft_consensus.cc:359] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.121184 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 84ad6c72a8704c148411881760848df4 (table default.Table_7814874203fc40af9ef07eeb2e11ebc4 [id=cc22b2daa8fb4eb1bae52b189d66b8fb]) successfully deleted
I20251026 02:56:33.121270 2381 raft_consensus.cc:385] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.121333 2381 raft_consensus.cc:740] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.121589 2377 tablet_replica.cc:333] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:33.121986 2375 tablet_bootstrap.cc:654] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.122334 2377 raft_consensus.cc:2243] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:33.123732 2377 pending_rounds.cc:70] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:33.124051 2377 pending_rounds.cc:77] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888520332349440 op_type: NO_OP noop_request { }
I20251026 02:56:33.124320 2377 raft_consensus.cc:2889] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:33.124558 2377 raft_consensus.cc:2272] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:33.122568 2381 consensus_queue.cc:260] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.125319 2381 ts_tablet_manager.cc:1434] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251026 02:56:33.127506 2378 tablet_bootstrap.cc:492] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:33.127698 2378 ts_tablet_manager.cc:1403] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20251026 02:56:33.128777 2381 tablet_bootstrap.cc:492] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:33.131374 2375 tablet_bootstrap.cc:492] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.131520 2375 ts_tablet_manager.cc:1403] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20251026 02:56:33.132045 2377 ts_tablet_manager.cc:1916] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:33.132275 2375 raft_consensus.cc:359] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.132452 2375 raft_consensus.cc:385] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.132508 2375 raft_consensus.cc:740] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.132730 2375 consensus_queue.cc:260] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.133157 2375 ts_tablet_manager.cc:1434] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:33.133411 2375 tablet_bootstrap.cc:492] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.134917 2375 tablet_bootstrap.cc:654] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.137693 2372 ts_tablet_manager.cc:1929] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:33.137828 2372 log.cc:1199] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/03da415f3a944a9a9679b3a45295ce35
I20251026 02:56:33.138249 2381 tablet_bootstrap.cc:654] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.138334 2372 ts_tablet_manager.cc:1950] T 03da415f3a944a9a9679b3a45295ce35 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:33.138595 2119 raft_consensus.cc:2468] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:33.138712 2378 raft_consensus.cc:359] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.138924 2378 raft_consensus.cc:385] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.139010 2378 raft_consensus.cc:740] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.139170 2376 ts_tablet_manager.cc:1929] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:33.139338 2376 log.cc:1199] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/98fe44aab33b40eba819386c3bffca46
I20251026 02:56:33.139261 2378 consensus_queue.cc:260] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.139807 2376 ts_tablet_manager.cc:1950] T 98fe44aab33b40eba819386c3bffca46 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:33.139853 2378 ts_tablet_manager.cc:1434] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20251026 02:56:33.144686 2378 tablet_bootstrap.cc:492] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:33.148950 2195 leader_election.cc:304] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.149688 3065 raft_consensus.cc:2804] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:33.149977 3065 raft_consensus.cc:697] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:33.150287 3065 consensus_queue.cc:237] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.150874 2375 tablet_bootstrap.cc:492] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.151019 2375 ts_tablet_manager.cc:1403] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20251026 02:56:33.151075 2378 tablet_bootstrap.cc:654] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.151708 2375 raft_consensus.cc:359] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.151882 2375 raft_consensus.cc:385] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.151942 2375 raft_consensus.cc:740] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.152151 2375 consensus_queue.cc:260] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.152657 2375 ts_tablet_manager.cc:1434] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:33.152937 2375 tablet_bootstrap.cc:492] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.154688 2375 tablet_bootstrap.cc:654] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.174417 2375 tablet_bootstrap.cc:492] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.174743 2375 ts_tablet_manager.cc:1403] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.015s
I20251026 02:56:33.175499 2375 raft_consensus.cc:359] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.175761 2375 raft_consensus.cc:385] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.175879 2375 raft_consensus.cc:740] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.176160 2375 consensus_queue.cc:260] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.176821 2375 ts_tablet_manager.cc:1434] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:33.177110 2375 tablet_bootstrap.cc:492] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.178005 1624 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
/tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.150.60:33729
--webserver_interface=127.1.150.60
--webserver_port=39671
--builtin_ntp_servers=127.1.150.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251026 02:56:33.186179 2375 tablet_bootstrap.cc:654] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.194744 2381 tablet_bootstrap.cc:492] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:33.195156 2381 ts_tablet_manager.cc:1403] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.002s
I20251026 02:56:33.195400 3007 raft_consensus.cc:493] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.195578 3007 raft_consensus.cc:515] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.196303 3007 leader_election.cc:290] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.197329 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3327d26a2c2c4d0b86d98151f5940bd7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:33.197523 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3327d26a2c2c4d0b86d98151f5940bd7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:33.197922 2053 leader_election.cc:343] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:33.198166 2053 leader_election.cc:343] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.198251 2053 leader_election.cc:304] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.198797 3007 raft_consensus.cc:2749] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:33.200049 2375 tablet_bootstrap.cc:492] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.200191 2375 ts_tablet_manager.cc:1403] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20251026 02:56:33.200922 2375 raft_consensus.cc:359] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.201093 2375 raft_consensus.cc:385] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.201148 2375 raft_consensus.cc:740] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.201344 2375 consensus_queue.cc:260] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.201833 2375 ts_tablet_manager.cc:1434] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:33.202092 2375 tablet_bootstrap.cc:492] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.218405 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_016812daf097412081aff793ad75f72a" } modify_external_catalogs: true
I20251026 02:56:33.218717 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_016812daf097412081aff793ad75f72a" } modify_external_catalogs: true
I20251026 02:56:33.230244 3007 raft_consensus.cc:493] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.230448 3007 raft_consensus.cc:515] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.233299 1654 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 5226d4516ce24ecda51dab35c86da132 (table default.Table_a6b94cbe15a34811a6a613f0d749a414 [id=c8435cef755d4de69c33f599df1c0f75]). Expected version 1 got 0
I20251026 02:56:33.234479 2377 ts_tablet_manager.cc:1929] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:33.234587 2377 log.cc:1199] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/aa399e6407fd4c4d9871d3d5ccf894ec
I20251026 02:56:33.234925 2377 ts_tablet_manager.cc:1950] T aa399e6407fd4c4d9871d3d5ccf894ec P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:33.234961 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 3791cb18fe1f4937badb4859d00647f3 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:33.235503 2114 raft_consensus.cc:2468] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:33.240921 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8c3d1109382545ffa157d6a3cb39f322" } modify_external_catalogs: true
I20251026 02:56:33.241214 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8c3d1109382545ffa157d6a3cb39f322" } modify_external_catalogs: true
I20251026 02:56:33.197189 2381 raft_consensus.cc:359] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.242947 2381 raft_consensus.cc:385] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.243083 2381 raft_consensus.cc:740] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.243268 2195 leader_election.cc:304] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.243382 2381 consensus_queue.cc:260] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.244316 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:33.245082 2053 leader_election.cc:343] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.245404 3007 leader_election.cc:290] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.246096 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 468) took 2249 ms. Trace:
I20251026 02:56:33.246222 2376 rpcz_store.cc:276] 1026 02:56:30.996565 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.000239 (+ 3674us) service_pool.cc:225] Handling call
1026 02:56:33.002106 (+2001867us) ts_tablet_manager.cc:1151] Deleting tablet 98fe44aab33b40eba819386c3bffca46
1026 02:56:33.002115 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.050682 (+ 48567us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.139023 (+ 88341us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.246018 (+106995us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2001612}
I20251026 02:56:33.247954 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 98fe44aab33b40eba819386c3bffca46 (table default.Table_481575c8f05747c2954c9c79a0ebde9c [id=91d789aff2464f73b60e03b52126a396]) successfully deleted
I20251026 02:56:33.243726 2962 raft_consensus.cc:2804] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:33.249048 2962 raft_consensus.cc:697] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:33.249423 2962 consensus_queue.cc:237] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.253283 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.255283 3190 tablet.cc:1722] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251026 02:56:33.256338 2053 leader_election.cc:343] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.256506 2053 leader_election.cc:304] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.257347 3007 raft_consensus.cc:2749] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:33.257912 2376 tablet_replica.cc:333] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:33.258208 2376 raft_consensus.cc:2243] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:33.258512 2376 raft_consensus.cc:2272] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:33.269613 2962 raft_consensus.cc:493] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:33.269711 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a956ebfb1eaa4d6fb1575b4165293193 has no consensus state
I20251026 02:56:33.269819 2962 raft_consensus.cc:515] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.270569 2962 leader_election.cc:290] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:33.271689 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a08efd37968846d2958e54831488235c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:33.271960 2119 raft_consensus.cc:2468] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:33.272790 2195 leader_election.cc:304] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:33.274158 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a08efd37968846d2958e54831488235c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:33.276024 2194 leader_election.cc:343] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.277160 2376 ts_tablet_manager.cc:1916] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:33.282634 3065 raft_consensus.cc:2804] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:33.282775 3065 raft_consensus.cc:493] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:33.282869 3065 raft_consensus.cc:3060] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.249433 2381 ts_tablet_manager.cc:1434] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.054s user 0.001s sys 0.001s
I20251026 02:56:33.287961 2381 tablet_bootstrap.cc:492] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:33.288998 3007 raft_consensus.cc:493] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.289149 3007 raft_consensus.cc:515] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.289743 3007 leader_election.cc:290] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.290627 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d2cc6b16d34331bb80c576c0cd2ce0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:33.291203 2053 leader_election.cc:343] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.291958 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d2cc6b16d34331bb80c576c0cd2ce0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.292390 2381 tablet_bootstrap.cc:654] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:33.292455 2053 leader_election.cc:343] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.292546 2053 leader_election.cc:304] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.292977 3007 raft_consensus.cc:2749] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:33.297053 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 490) took 2172 ms. Trace:
I20251026 02:56:33.297008 3191 tablet.cc:1722] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:33.297135 2372 rpcz_store.cc:276] 1026 02:56:31.124880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.126809 (+ 1929us) service_pool.cc:225] Handling call
1026 02:56:32.850241 (+1723432us) ts_tablet_manager.cc:1151] Deleting tablet 03da415f3a944a9a9679b3a45295ce35
1026 02:56:32.850252 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:32.892536 (+ 42284us) spinlock_profiling.cc:243] Waited 41.6 ms on lock 0x616000043f54. stack: 00007f8f76bbd9f6 00007f8f76bbd7b0 00007f8f75b993ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8f82a83100 00007f8f83dd0ae3 00007f8f8698d287 00007f8f869c02d4 00007f8f869a0d70 00007f8f869a0afd 0000000000a8867c 00007f8f76c03d7f 00007f8f76c0704b 00007f8f76c06ec8
1026 02:56:33.037107 (+144571us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.137545 (+100438us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.297038 (+159493us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57684,"spinlock_wait_cycles":95794560,"tablet-delete.queue_time_us":1723222}
I20251026 02:56:33.297489 2372 tablet_replica.cc:333] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:33.297505 2378 tablet_bootstrap.cc:492] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:33.297664 2378 ts_tablet_manager.cc:1403] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.153s user 0.026s sys 0.066s
I20251026 02:56:33.297680 2372 raft_consensus.cc:2243] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:33.297832 2372 raft_consensus.cc:2272] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:33.298424 2372 ts_tablet_manager.cc:1916] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:33.298565 2378 raft_consensus.cc:359] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.298739 2378 raft_consensus.cc:385] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.298813 2378 raft_consensus.cc:740] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.299023 2378 consensus_queue.cc:260] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.300299 2378 ts_tablet_manager.cc:1434] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251026 02:56:33.300468 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 03da415f3a944a9a9679b3a45295ce35 (table default.Table_828c16a96a47489f835a891f34d707b1 [id=432234d273ca4c9ea53c7a078c274637]) successfully deleted
I20251026 02:56:33.300609 2378 tablet_bootstrap.cc:492] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:33.302623 2378 tablet_bootstrap.cc:654] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:33.302980 2194 consensus_peers.cc:597] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:33.304199 2381 tablet_bootstrap.cc:492] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:33.304368 2381 ts_tablet_manager.cc:1403] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.005s
I20251026 02:56:33.305110 2381 raft_consensus.cc:359] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.305388 2381 raft_consensus.cc:385] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.305442 2381 raft_consensus.cc:740] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.305698 2378 tablet_bootstrap.cc:492] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:33.305629 2381 consensus_queue.cc:260] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.305840 2378 ts_tablet_manager.cc:1403] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20251026 02:56:33.306108 2381 ts_tablet_manager.cc:1434] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:33.306414 2381 tablet_bootstrap.cc:492] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:33.306568 2378 raft_consensus.cc:359] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.306736 2378 raft_consensus.cc:385] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.306818 2378 raft_consensus.cc:740] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.307463 2378 consensus_queue.cc:260] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.308014 2381 tablet_bootstrap.cc:654] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.309288 2375 tablet_bootstrap.cc:654] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.311096 3164 raft_consensus.cc:493] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.311252 3164 raft_consensus.cc:515] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.311991 3164 leader_election.cc:290] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.312891 2378 ts_tablet_manager.cc:1434] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20251026 02:56:33.312976 1916 leader_election.cc:343] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 86d8e1e21c5f440d8baf76c357d01ad2
I20251026 02:56:33.314392 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86d8e1e21c5f440d8baf76c357d01ad2" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.314651 2378 tablet_bootstrap.cc:492] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:33.314901 1915 leader_election.cc:304] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:33.315168 3164 raft_consensus.cc:2749] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:33.317509 2378 tablet_bootstrap.cc:654] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.323175 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 425) took 4619 ms. Trace:
I20251026 02:56:33.323355 2377 rpcz_store.cc:276] 1026 02:56:28.703882 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.703942 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:33.121540 (+4417598us) ts_tablet_manager.cc:1151] Deleting tablet aa399e6407fd4c4d9871d3d5ccf894ec
1026 02:56:33.121552 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.231170 (+109618us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.234382 (+ 3212us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.323141 (+ 88759us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4378595}
I20251026 02:56:33.323946 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 436) took 4557 ms. Trace:
I20251026 02:56:33.324076 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet aa399e6407fd4c4d9871d3d5ccf894ec (table default.Table_65488c1f07e542b5973be5cfd122787c [id=ce3610a9d6814ec783ae7503e58ef3ad]) successfully deleted
I20251026 02:56:33.326416 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f8baefeba2b54cb39affb2d4f1b0bef8" } modify_external_catalogs: true
I20251026 02:56:33.325474 2375 tablet_bootstrap.cc:492] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.326674 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f8baefeba2b54cb39affb2d4f1b0bef8" } modify_external_catalogs: true
I20251026 02:56:33.326730 2375 ts_tablet_manager.cc:1403] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.125s user 0.002s sys 0.002s
I20251026 02:56:33.324038 2377 rpcz_store.cc:276] 1026 02:56:28.766116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.774762 (+ 8646us) service_pool.cc:225] Handling call
1026 02:56:33.323767 (+4549005us) ts_tablet_manager.cc:1151] Deleting tablet 41b7b0c36ba748f0a1565892dbe98a78
1026 02:56:33.323794 (+ 27us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.323924 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4512982}
I20251026 02:56:33.327510 2375 raft_consensus.cc:359] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.327858 2377 tablet_replica.cc:333] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539: stopping tablet replica
W20251026 02:56:33.328003 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 41b7b0c36ba748f0a1565892dbe98a78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41b7b0c36ba748f0a1565892dbe98a78 already in progress: creating tablet
I20251026 02:56:33.328130 2377 raft_consensus.cc:2243] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
W20251026 02:56:33.325997 2053 consensus_peers.cc:597] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:33.328538 2377 pending_rounds.cc:70] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:33.328580 2377 pending_rounds.cc:77] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888505934307328 op_type: NO_OP noop_request { }
I20251026 02:56:33.328661 2377 raft_consensus.cc:2889] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:33.328739 2377 raft_consensus.cc:2272] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:33.330180 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 3e38f282e8414b7db2a6d5f4a8933e84 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:33.331221 2195 consensus_peers.cc:597] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0efdcc36ff5b47379e0a76512e05d379. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:33.331620 2377 ts_tablet_manager.cc:1916] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:33.332366 3065 raft_consensus.cc:515] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.332963 3065 leader_election.cc:290] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:33.333410 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet e62064fbdffb4954a3a1227efd6327ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:33.340267 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a08efd37968846d2958e54831488235c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:33.340924 2194 leader_election.cc:343] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.341230 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a08efd37968846d2958e54831488235c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:33.341411 2114 raft_consensus.cc:3060] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.343644 2378 tablet_bootstrap.cc:492] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:33.343832 2378 ts_tablet_manager.cc:1403] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.017s
I20251026 02:56:33.327662 2375 raft_consensus.cc:385] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.346058 2375 raft_consensus.cc:740] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.348870 2375 consensus_queue.cc:260] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.351603 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 865fbd674db34f668fb5a2f5836a9b43 (table default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa]). Expected version 1 got 0
I20251026 02:56:33.353852 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet a70d70abc3a34dac8cf9a747903fa78c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:33.354012 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b3182aeed9424bcb9d78331e40f7867f" } new_table_name: "default.Table_0a47e554e1874269a6d13aef19c29678" modify_external_catalogs: true
I20251026 02:56:33.356452 2376 ts_tablet_manager.cc:1929] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:33.356596 2376 log.cc:1199] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/22e4a433801a4cb58062d3e76c760b69
I20251026 02:56:33.357060 2376 ts_tablet_manager.cc:1950] T 22e4a433801a4cb58062d3e76c760b69 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:33.357543 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6b9199ff4e884b4891446da6e6e6195d" } modify_external_catalogs: true
I20251026 02:56:33.357743 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6b9199ff4e884b4891446da6e6e6195d" } modify_external_catalogs: true
I20251026 02:56:33.358487 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 903e65fa8c214dc9a50cb97cab101d22
I20251026 02:56:33.358549 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 548a7f9628ac4623b2c2f723f83c1e45
I20251026 02:56:33.358878 2375 ts_tablet_manager.cc:1434] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20251026 02:56:33.359004 1661 catalog_manager.cc:6719] ReplaceTablet: tablet 903e65fa8c214dc9a50cb97cab101d22 deleted and replaced by tablet 5f8d8c4fbd474517961ea831806e6503
I20251026 02:56:33.359182 2375 tablet_bootstrap.cc:492] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.359172 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet 1fb2ff87cf0248a0a4a77f2733ef867b from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:33.359360 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7d238b86df1246ffa4b5d4bb9af0f387"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.359930 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d238b86df1246ffa4b5d4bb9af0f387 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.360931 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 548a7f9628ac4623b2c2f723f83c1e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:33.361150 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 903e65fa8c214dc9a50cb97cab101d22 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f8d8c4fbd474517961ea831806e6503) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:33.361245 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a821ed8a686c452a975acb96b5c74778
I20251026 02:56:33.361361 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 548a7f9628ac4623b2c2f723f83c1e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.361877 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ec68beb934274468a4f028b3dbaf2cc0" } modify_external_catalogs: true
I20251026 02:56:33.362063 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ec68beb934274468a4f028b3dbaf2cc0" } modify_external_catalogs: true
I20251026 02:56:33.362474 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet a821ed8a686c452a975acb96b5c74778 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.362860 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 903e65fa8c214dc9a50cb97cab101d22 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f8d8c4fbd474517961ea831806e6503) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.362922 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet a821ed8a686c452a975acb96b5c74778 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:33.363600 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 903e65fa8c214dc9a50cb97cab101d22 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f8d8c4fbd474517961ea831806e6503) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.363822 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 548a7f9628ac4623b2c2f723f83c1e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.364499 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet a821ed8a686c452a975acb96b5c74778 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.364984 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7a4c0546cb2e4efdb10c4f630a115607" } new_table_name: "default.Table_b56f9f35731e4bc4be24623e8e79fe5e" modify_external_catalogs: true
I20251026 02:56:33.371497 2375 tablet_bootstrap.cc:654] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.372965 2378 raft_consensus.cc:359] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.373323 2378 raft_consensus.cc:385] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.373571 2378 raft_consensus.cc:740] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.373960 2378 consensus_queue.cc:260] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.373833 2381 tablet_bootstrap.cc:492] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
W20251026 02:56:33.374815 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3aea01a248b345cf81d4d9862744eb78 consensus state has no leader
I20251026 02:56:33.374827 2381 ts_tablet_manager.cc:1403] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.011s
I20251026 02:56:33.375176 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet d0f1d6967931482e9de40b0844da6315 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:33.375876 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_43ea6a50a51e416da1649a35e72a7283"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:33.376261 2381 raft_consensus.cc:359] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.376434 2381 raft_consensus.cc:385] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.376490 2381 raft_consensus.cc:740] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
W20251026 02:56:33.376472 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43ea6a50a51e416da1649a35e72a7283 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.376721 2381 consensus_queue.cc:260] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:33.378453 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c740ab6481e4481b95878c3a185129d has no consensus state
I20251026 02:56:33.378818 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9c462c3663254348a3d6f661a26cd0bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.379348 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c462c3663254348a3d6f661a26cd0bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.380358 3007 raft_consensus.cc:493] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.380522 3007 raft_consensus.cc:515] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.381155 3007 leader_election.cc:290] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.383030 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a378e913112c443db869016e3b98fd0e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:33.383845 2053 leader_election.cc:343] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.384379 2381 ts_tablet_manager.cc:1434] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251026 02:56:33.382817 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a378e913112c443db869016e3b98fd0e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:33.390429 2053 leader_election.cc:343] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.390575 2053 leader_election.cc:304] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.390846 3007 raft_consensus.cc:2749] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:33.390846 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a6b94cbe15a34811a6a613f0d749a414" } new_table_name: "default.Table_40e6c3d6d5364f56a9517c1bfe79f773" modify_external_catalogs: true
I20251026 02:56:33.393579 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_051c1782e1cd4f8caa8dbefeddb868f9" } new_table_name: "default.Table_103c3d38235f4771a4cad7b79143a809" modify_external_catalogs: true
I20251026 02:56:33.394649 2381 tablet_bootstrap.cc:492] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:33.395099 2962 raft_consensus.cc:493] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.395229 2962 raft_consensus.cc:515] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.395887 2962 leader_election.cc:290] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:33.396579 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63a63b796191465aab9a63d3ffb0554d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:33.396812 2118 raft_consensus.cc:2468] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:33.397255 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63a63b796191465aab9a63d3ffb0554d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:33.397755 2195 leader_election.cc:304] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
W20251026 02:56:33.398160 2194 leader_election.cc:343] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.398427 2378 ts_tablet_manager.cc:1434] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.054s user 0.003s sys 0.000s
I20251026 02:56:33.398538 2962 raft_consensus.cc:2804] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:33.398614 2962 raft_consensus.cc:493] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:33.398671 2962 raft_consensus.cc:3060] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.403652 2378 tablet_bootstrap.cc:492] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:33.405771 2378 tablet_bootstrap.cc:654] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.406973 2381 tablet_bootstrap.cc:654] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.407693 2378 tablet_bootstrap.cc:492] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:33.407889 2378 ts_tablet_manager.cc:1403] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251026 02:56:33.409392 2375 tablet_bootstrap.cc:492] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.409566 2375 ts_tablet_manager.cc:1403] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20251026 02:56:33.410535 2375 raft_consensus.cc:359] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.410710 2375 raft_consensus.cc:385] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.410756 2375 raft_consensus.cc:740] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.410969 2375 consensus_queue.cc:260] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.411458 2375 ts_tablet_manager.cc:1434] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:33.411768 2375 tablet_bootstrap.cc:492] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.412653 3190 tablet.cc:1722] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:33.413580 2375 tablet_bootstrap.cc:654] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:33.415153 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31edacb952514c8aa25c79f509869e8f has no consensus state
I20251026 02:56:33.417794 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fda5e7b00a8a4e9ba95382d569d8c421"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.418563 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fda5e7b00a8a4e9ba95382d569d8c421 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.420167 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c76a8a4746f9463eb166591b7c7b5713"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:33.421700 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet ca4611c0a494425ca50d599b310872c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:33.421797 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 469) took 2291 ms. Trace:
I20251026 02:56:33.421932 2376 rpcz_store.cc:276] 1026 02:56:31.129955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.130001 (+ 46us) service_pool.cc:225] Handling call
1026 02:56:33.246592 (+2116591us) ts_tablet_manager.cc:1151] Deleting tablet 22e4a433801a4cb58062d3e76c760b69
1026 02:56:33.257734 (+ 11142us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.332432 (+ 74698us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.356294 (+ 23862us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.421747 (+ 65453us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2116342}
W20251026 02:56:33.422950 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c76a8a4746f9463eb166591b7c7b5713 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.423089 2376 tablet_replica.cc:333] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:33.423346 2376 raft_consensus.cc:2243] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:33.423529 2376 raft_consensus.cc:2272] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:33.408797 2378 raft_consensus.cc:359] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.424194 2378 raft_consensus.cc:385] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.424477 2378 raft_consensus.cc:740] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.424707 2378 consensus_queue.cc:260] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.424907 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 22e4a433801a4cb58062d3e76c760b69 (table default.Table_f7c216f3ebd34c718ceef042a435b4cd [id=b4c0611288a949378eed16d32b5170b9]) successfully deleted
I20251026 02:56:33.424166 2376 ts_tablet_manager.cc:1916] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:33.429217 2378 ts_tablet_manager.cc:1434] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251026 02:56:33.431746 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (table default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5]). Expected version 1 got 0
I20251026 02:56:33.433905 2378 tablet_bootstrap.cc:492] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:33.436478 2114 raft_consensus.cc:2468] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:33.436689 2378 tablet_bootstrap.cc:654] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.436962 2962 raft_consensus.cc:515] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.437892 2962 leader_election.cc:290] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:33.438097 1658 catalog_manager.cc:5649] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:33.438647 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 94886ad6d6bb45f39261725169774ac9 (table default.Table_d23ac2c9657b4adeaac9545d5bc9ae41 [id=9e897d33e6374ad4bc3c8acaae512645]). Expected version 1 got 0
I20251026 02:56:33.438632 2195 leader_election.cc:304] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.439695 2372 ts_tablet_manager.cc:1929] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:33.439640 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63a63b796191465aab9a63d3ffb0554d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:33.439801 2372 log.cc:1199] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/ca4611c0a494425ca50d599b310872c8
I20251026 02:56:33.439862 2114 raft_consensus.cc:3060] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.440109 2372 ts_tablet_manager.cc:1950] T ca4611c0a494425ca50d599b310872c8 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:33.444547 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63a63b796191465aab9a63d3ffb0554d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:33.444770 2962 raft_consensus.cc:2804] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:33.444957 2962 raft_consensus.cc:697] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:33.445307 2962 consensus_queue.cc:237] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:33.447757 2194 leader_election.cc:343] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.448625 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d9351d4a858248d8813e9df05f900010"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:33.450947 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.451189 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet cfc3aed64ad0439cb22fd6815c973d87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b77bb6b4722449c9699b548a9ef6314) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.451375 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet efd751ee4b1d433198dcd00c081f648d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.451531 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 3327d26a2c2c4d0b86d98151f5940bd7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7177931353164b2aa82c00b2807ccd63) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.451712 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 03da415f3a944a9a9679b3a45295ce35 with delete_type TABLET_DATA_DELETED (replaced by tablet a6f9ae327aa4401baf66a737eaa93cf6) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.455294 1976 raft_consensus.cc:3060] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:33.458895 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9351d4a858248d8813e9df05f900010 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.460547 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1fb2ff87cf0248a0a4a77f2733ef867b
I20251026 02:56:33.461138 1656 catalog_manager.cc:6719] ReplaceTablet: tablet 1fb2ff87cf0248a0a4a77f2733ef867b deleted and replaced by tablet ce11081bf85f44e8bf1f92ce0f4bf152
I20251026 02:56:33.461524 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fb2ff87cf0248a0a4a77f2733ef867b with delete_type TABLET_DATA_DELETED (replaced by tablet ce11081bf85f44e8bf1f92ce0f4bf152) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:33.461551 3164 raft_consensus.cc:493] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.461942 3164 raft_consensus.cc:515] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.462656 1661 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 358e805370ef410e9d65ee92a8d04f34 (table default.Table_71583d73a49142ae9fec31193fb53542 [id=e86d9fc892a8409590c555e25b27a12f]). Expected version 1 got 0
I20251026 02:56:33.462810 3164 leader_election.cc:290] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.464075 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fb2ff87cf0248a0a4a77f2733ef867b with delete_type TABLET_DATA_DELETED (replaced by tablet ce11081bf85f44e8bf1f92ce0f4bf152) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.464387 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1a2522801364693be0bda6fa45a697e" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.464643 2263 raft_consensus.cc:2468] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:33.464859 3065 consensus_queue.cc:1048] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:33.465775 1916 leader_election.cc:343] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: b1a2522801364693be0bda6fa45a697e
I20251026 02:56:33.466101 1915 leader_election.cc:304] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:33.466670 3164 raft_consensus.cc:2804] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:33.466791 3164 raft_consensus.cc:493] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:33.466931 3164 raft_consensus.cc:3060] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.469736 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fb2ff87cf0248a0a4a77f2733ef867b with delete_type TABLET_DATA_DELETED (replaced by tablet ce11081bf85f44e8bf1f92ce0f4bf152) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.473014 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet aa399e6407fd4c4d9871d3d5ccf894ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.473299 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 634d913c98c842a49d97a54f23752d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.485095 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 34c6a7c774044411a4733b115080c4c6
I20251026 02:56:33.485298 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3e38f282e8414b7db2a6d5f4a8933e84
I20251026 02:56:33.485363 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 892a5fab2af1432eb31d92d9859ea2a0
I20251026 02:56:33.485781 1663 catalog_manager.cc:6719] ReplaceTablet: tablet 3e38f282e8414b7db2a6d5f4a8933e84 deleted and replaced by tablet 2e0b929f603b4c25b5b971ee9db683e8
I20251026 02:56:33.486129 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet d0f1d6967931482e9de40b0844da6315
I20251026 02:56:33.486665 1655 catalog_manager.cc:6719] ReplaceTablet: tablet d0f1d6967931482e9de40b0844da6315 deleted and replaced by tablet 1360fe4278f24f31b4ea3eb28d5abf5d
I20251026 02:56:33.487376 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e38f282e8414b7db2a6d5f4a8933e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e0b929f603b4c25b5b971ee9db683e8) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.487675 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 892a5fab2af1432eb31d92d9859ea2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.487975 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet d0f1d6967931482e9de40b0844da6315 with delete_type TABLET_DATA_DELETED (replaced by tablet 1360fe4278f24f31b4ea3eb28d5abf5d) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.488253 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 34c6a7c774044411a4733b115080c4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:33.488294 1648 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 4db275efdd344817b11a2168675cf95e (table default.Table_c30cfa52b6524323bda478613d74178b [id=1fd74186ea10498599435dccd7330d25], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4db275efdd344817b11a2168675cf95e consensus state has no leader
I20251026 02:56:33.491533 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e38f282e8414b7db2a6d5f4a8933e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e0b929f603b4c25b5b971ee9db683e8) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:33.491832 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 892a5fab2af1432eb31d92d9859ea2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:33.491561 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet d0f1d6967931482e9de40b0844da6315 with delete_type TABLET_DATA_DELETED (replaced by tablet 1360fe4278f24f31b4ea3eb28d5abf5d) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:33.492596 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 34c6a7c774044411a4733b115080c4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:33.493196 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e38f282e8414b7db2a6d5f4a8933e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e0b929f603b4c25b5b971ee9db683e8) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.493716 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 892a5fab2af1432eb31d92d9859ea2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.494104 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 34c6a7c774044411a4733b115080c4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.494736 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet d0f1d6967931482e9de40b0844da6315 with delete_type TABLET_DATA_DELETED (replaced by tablet 1360fe4278f24f31b4ea3eb28d5abf5d) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.496438 1978 raft_consensus.cc:3060] T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.498999 3006 raft_consensus.cc:493] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.499158 3006 raft_consensus.cc:515] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.499810 3006 leader_election.cc:290] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.500597 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7ffdaeec2240dcbe4cafdb96073a7d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.500689 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7ffdaeec2240dcbe4cafdb96073a7d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:33.501278 2053 leader_election.cc:343] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:33.501528 2053 leader_election.cc:343] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.501644 2053 leader_election.cc:304] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.502137 3006 raft_consensus.cc:2749] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:33.504052 3206 tablet.cc:1722] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:33.505616 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_afa5656eb2a74f6ca79f85270f7c97eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.506258 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afa5656eb2a74f6ca79f85270f7c97eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.508980 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d1fa467b0a5a410aa8905dfabb5b5ea1" } new_table_name: "default.Table_a345a2617d614092bedde12cb0020ce8" modify_external_catalogs: true
I20251026 02:56:33.512414 3065 raft_consensus.cc:493] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.512631 3065 raft_consensus.cc:515] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.513439 3065 leader_election.cc:290] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:33.514617 2119 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "634d913c98c842a49d97a54f23752d26" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:33.514874 2119 raft_consensus.cc:2468] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:33.515451 2195 leader_election.cc:304] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:33.515996 3065 raft_consensus.cc:2804] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:33.516136 3065 raft_consensus.cc:493] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:33.516290 3065 raft_consensus.cc:3060] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.519618 1957 tablet_service.cc:1505] Processing CreateTablet for tablet b4005a095c594469b8d8105a0679ff45 (DEFAULT_TABLE table=default.Table_0635cec42beb4731b5362de5a1e6b806 [id=f88198867c24439f85e045c77f752e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.520607 2226 tablet_service.cc:1505] Processing CreateTablet for tablet b4005a095c594469b8d8105a0679ff45 (DEFAULT_TABLE table=default.Table_0635cec42beb4731b5362de5a1e6b806 [id=f88198867c24439f85e045c77f752e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.521121 3163 raft_consensus.cc:493] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.521250 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "634d913c98c842a49d97a54f23752d26" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:33.521343 3163 raft_consensus.cc:515] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.521473 2091 tablet_service.cc:1505] Processing CreateTablet for tablet b4005a095c594469b8d8105a0679ff45 (DEFAULT_TABLE table=default.Table_0635cec42beb4731b5362de5a1e6b806 [id=f88198867c24439f85e045c77f752e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.522033 3163 leader_election.cc:290] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.522033 1943 tablet_service.cc:1505] Processing CreateTablet for tablet f15000847df44f1ebebad3ca2c2b8046 (DEFAULT_TABLE table=default.Table_5edc3cfde43b432ca349a34eb81cc365 [id=d4dadc9f79c644b4bcdca125810ce6fa]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:33.523002 1916 leader_election.cc:343] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: dbfe80d388644d0488cb2431a334c908
I20251026 02:56:33.523015 2096 tablet_service.cc:1505] Processing CreateTablet for tablet f15000847df44f1ebebad3ca2c2b8046 (DEFAULT_TABLE table=default.Table_5edc3cfde43b432ca349a34eb81cc365 [id=d4dadc9f79c644b4bcdca125810ce6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.523118 2236 tablet_service.cc:1505] Processing CreateTablet for tablet f15000847df44f1ebebad3ca2c2b8046 (DEFAULT_TABLE table=default.Table_5edc3cfde43b432ca349a34eb81cc365 [id=d4dadc9f79c644b4bcdca125810ce6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.523751 2229 tablet_service.cc:1505] Processing CreateTablet for tablet cedff3287c7b4a07a81462e8c4b44970 (DEFAULT_TABLE table=default.Table_f5e1fc5d0a604a82be231f8b70013315 [id=8a281c625796438b8671aca9e4b818ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.524044 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbfe80d388644d0488cb2431a334c908" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:33.524725 1915 leader_election.cc:343] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: Tablet not RUNNING: STOPPED
I20251026 02:56:33.524919 1915 leader_election.cc:304] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:33.525362 3163 raft_consensus.cc:2749] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:33.525465 2194 leader_election.cc:343] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.526533 1950 tablet_service.cc:1505] Processing CreateTablet for tablet cedff3287c7b4a07a81462e8c4b44970 (DEFAULT_TABLE table=default.Table_f5e1fc5d0a604a82be231f8b70013315 [id=8a281c625796438b8671aca9e4b818ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.527346 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 94886ad6d6bb45f39261725169774ac9 (table default.Table_d23ac2c9657b4adeaac9545d5bc9ae41 [id=9e897d33e6374ad4bc3c8acaae512645]). Expected version 1 got 0
I20251026 02:56:33.527649 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet dba102838945453cbf44e1cfe505a52c (table default.Table_8ffce6cf6b2d44aa8a9459e834de1d80 [id=92310ee99b5c428c9773b0f7a5462a4d]). Expected version 1 got 0
I20251026 02:56:33.528803 1663 catalog_manager.cc:5649] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:33.529516 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 352a284902c94f088e2e6ae006162db0 (table default.Table_6cd4ee82e9ec406f971669b7f0722677 [id=414375003cd44b5c8c9f0249b339c28a]). Expected version 1 got 0
I20251026 02:56:33.530920 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet be1c0948c0ae4563b7cb644521914cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.532150 3191 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:33.532816 2375 tablet_bootstrap.cc:492] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.532943 2375 ts_tablet_manager.cc:1403] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.121s user 0.007s sys 0.001s
I20251026 02:56:33.533664 2375 raft_consensus.cc:359] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.533823 2375 raft_consensus.cc:385] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.533864 2375 raft_consensus.cc:740] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.534051 2375 consensus_queue.cc:260] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.534274 2377 ts_tablet_manager.cc:1929] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:33.534417 2377 log.cc:1199] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/dbfe80d388644d0488cb2431a334c908
I20251026 02:56:33.534494 2375 ts_tablet_manager.cc:1434] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 02:56:33.534754 2375 tablet_bootstrap.cc:492] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.534842 2377 ts_tablet_manager.cc:1950] T dbfe80d388644d0488cb2431a334c908 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:33.535063 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4005a095c594469b8d8105a0679ff45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.535602 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cedff3287c7b4a07a81462e8c4b44970. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.537345 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f15000847df44f1ebebad3ca2c2b8046. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.537427 2093 tablet_service.cc:1505] Processing CreateTablet for tablet cedff3287c7b4a07a81462e8c4b44970 (DEFAULT_TABLE table=default.Table_f5e1fc5d0a604a82be231f8b70013315 [id=8a281c625796438b8671aca9e4b818ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.537771 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cedff3287c7b4a07a81462e8c4b44970. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.539100 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 47037db3602246a2ab6224557286162f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.540318 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet dbf2a1ce78154415b23ab6d0efbc1f52 with delete_type TABLET_DATA_DELETED (replaced by tablet f51f8fef342b46abafff5bf12dba7184) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.544143 3166 raft_consensus.cc:493] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.544369 3166 raft_consensus.cc:515] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.545116 3166 leader_election.cc:290] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:33.546440 1916 leader_election.cc:343] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: a70d70abc3a34dac8cf9a747903fa78c
I20251026 02:56:33.546886 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 4ed205c1800243ed9830d720ac85de2f (DEFAULT_TABLE table=default.Table_082dd939577943e5ae1d59dea699d4ca [id=02058fa4c4cc482d8d0de7dc04d274a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.547652 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 17d2cc6b16d34331bb80c576c0cd2ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.548076 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet ce922040298d4952af13652eeda7a234 with delete_type TABLET_DATA_DELETED (replaced by tablet e830f1aa1270420c9b28a0491a1a0960) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.548583 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 44af74033ccf4d048475f12d164938f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.550248 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a70d70abc3a34dac8cf9a747903fa78c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.550524 2263 raft_consensus.cc:2468] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:33.552474 2375 tablet_bootstrap.cc:654] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.552445 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2d0444a3af524d67ac524f71f4d70f11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:33.553042 2114 raft_consensus.cc:2468] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:33.553088 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d0444a3af524d67ac524f71f4d70f11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.553840 2195 leader_election.cc:304] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.554677 3028 raft_consensus.cc:2804] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:33.554659 1915 leader_election.cc:304] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:33.554844 3028 raft_consensus.cc:697] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:33.554808 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 4ed205c1800243ed9830d720ac85de2f (DEFAULT_TABLE table=default.Table_082dd939577943e5ae1d59dea699d4ca [id=02058fa4c4cc482d8d0de7dc04d274a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.555164 3163 raft_consensus.cc:2804] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:33.555267 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed205c1800243ed9830d720ac85de2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.555160 3028 consensus_queue.cc:237] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.555267 3163 raft_consensus.cc:493] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:33.555382 3163 raft_consensus.cc:3060] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.556526 2934 raft_consensus.cc:493] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.556650 2934 raft_consensus.cc:515] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.557179 2934 leader_election.cc:290] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.558494 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 4ed205c1800243ed9830d720ac85de2f (DEFAULT_TABLE table=default.Table_082dd939577943e5ae1d59dea699d4ca [id=02058fa4c4cc482d8d0de7dc04d274a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.558884 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed205c1800243ed9830d720ac85de2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.560102 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce1ab21ddf8e45bc9063b63db9c1b97f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.564383 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce1ab21ddf8e45bc9063b63db9c1b97f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:33.564734 2053 leader_election.cc:343] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.565069 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 3efafaac6c634d088396d56c8debd990 (DEFAULT_TABLE table=default.Table_44853e936f824114a26a5c01ae8543d3 [id=0558cab549f94833aed58ea430e03566]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:33.566274 2194 consensus_peers.cc:597] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:33.567262 2053 leader_election.cc:343] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.567404 2053 leader_election.cc:304] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.567811 2934 raft_consensus.cc:2749] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:33.573325 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6979455632e54d2081065954771f34d3 consensus state has no leader
I20251026 02:56:33.573836 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1818605e0bc548938337de5d5fc7a781" } modify_external_catalogs: true
I20251026 02:56:33.574012 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1818605e0bc548938337de5d5fc7a781" } modify_external_catalogs: true
I20251026 02:56:33.574388 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_42d878cfb9464b9a8403399260da8d43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.574944 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42d878cfb9464b9a8403399260da8d43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.576175 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c4c451a485f24525b3dbb3f3626a01e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:33.576440 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1e459a892e604d56aeb212131b5a624f" } modify_external_catalogs: true
I20251026 02:56:33.576608 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1e459a892e604d56aeb212131b5a624f" } modify_external_catalogs: true
W20251026 02:56:33.576730 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4c451a485f24525b3dbb3f3626a01e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.579090 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8db1d3c49d8d44ffa6c3160fa1720e20" } new_table_name: "default.Table_760b799491d7426a86aa1dc3c76fab31" modify_external_catalogs: true
I20251026 02:56:33.579411 2934 raft_consensus.cc:493] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.579555 2934 raft_consensus.cc:515] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.580153 2934 leader_election.cc:290] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.580593 2263 raft_consensus.cc:1275] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Refusing update from remote peer 0b6b0a6c4e144982b67980118cddba48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:33.581451 2934 consensus_queue.cc:1048] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:33.582389 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be1c0948c0ae4563b7cb644521914cb9" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.582525 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be1c0948c0ae4563b7cb644521914cb9" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:33.583176 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet 863ce9e1adb444b7b1399b968ea79bb4 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:33.583125 2053 leader_election.cc:343] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:33.583451 2053 leader_election.cc:343] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.583551 2053 leader_election.cc:304] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.584576 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d23ac2c9657b4adeaac9545d5bc9ae41" } new_table_name: "default.Table_73789469dbaa4885b8566a3aa0877893" modify_external_catalogs: true
I20251026 02:56:33.583523 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet a378e913112c443db869016e3b98fd0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.586812 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 3efafaac6c634d088396d56c8debd990 (DEFAULT_TABLE table=default.Table_44853e936f824114a26a5c01ae8543d3 [id=0558cab549f94833aed58ea430e03566]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.587247 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efafaac6c634d088396d56c8debd990. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.587910 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ed836bb7519e4b0290e6547fa57128bc
I20251026 02:56:33.588997 3065 raft_consensus.cc:515] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.590565 3065 leader_election.cc:290] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:33.589905 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 439) took 4652 ms. Trace:
I20251026 02:56:33.592047 1662 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 8688a0bf946841deb5d88c931538b129 (table default.Table_a6e8714ddccb4139adb6c8320e06500b [id=bb34b75c6fcb4ea6b201d05be4982a75]). Expected version 1 got 0
I20251026 02:56:33.592327 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "634d913c98c842a49d97a54f23752d26" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:33.592233 2377 rpcz_store.cc:276] 1026 02:56:28.937215 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.937922 (+ 707us) service_pool.cc:225] Handling call
1026 02:56:33.327786 (+4389864us) ts_tablet_manager.cc:1151] Deleting tablet dbfe80d388644d0488cb2431a334c908
1026 02:56:33.327801 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.432912 (+105111us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.534091 (+101179us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.589859 (+ 55768us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4364482}
I20251026 02:56:33.592514 2114 raft_consensus.cc:3060] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.592686 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 438) took 4655 ms. Trace:
I20251026 02:56:33.592825 2377 rpcz_store.cc:276] 1026 02:56:28.937109 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.937899 (+ 790us) service_pool.cc:225] Handling call
1026 02:56:33.592544 (+4654645us) ts_tablet_manager.cc:1151] Deleting tablet dfd3e23d83a84648bd11cd258377aecb
1026 02:56:33.592554 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.592655 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4628573}
I20251026 02:56:33.593353 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 437) took 4656 ms. Trace:
I20251026 02:56:33.593669 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet ed836bb7519e4b0290e6547fa57128bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.593832 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "634d913c98c842a49d97a54f23752d26" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:33.594247 2085 tablet_service.cc:1505] Processing CreateTablet for tablet e044af9739364ecdbbaddc115c09bfab (DEFAULT_TABLE table=default.Table_6b7a7d49a4cf4c7084d664fa624c3da4 [id=5e5fa8c80fd24402a3e9bd94b263bff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.594620 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e044af9739364ecdbbaddc115c09bfab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.594893 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 58d60bf2a9e546e6a9474a5af6d5273a (DEFAULT_TABLE table=default.Table_cef54300186c47eca175dff1adf385db [id=6b5fd8a21a8f42718d77deb53f71f414]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.595445 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet ed836bb7519e4b0290e6547fa57128bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:33.595924 1960 tablet_service.cc:1505] Processing CreateTablet for tablet e044af9739364ecdbbaddc115c09bfab (DEFAULT_TABLE table=default.Table_6b7a7d49a4cf4c7084d664fa624c3da4 [id=5e5fa8c80fd24402a3e9bd94b263bff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.596611 1949 tablet_service.cc:1505] Processing CreateTablet for tablet 1ffead7ca2d946c78aa4b8a602865fc8 (DEFAULT_TABLE table=default.Table_ac31a485be77410ca1fafabe643f37a2 [id=1a611c5db865445e840b4e227852e819]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:33.597355 2194 leader_election.cc:343] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.597657 2377 rpcz_store.cc:276] 1026 02:56:28.937011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:28.937871 (+ 860us) service_pool.cc:225] Handling call
1026 02:56:33.593244 (+4655373us) ts_tablet_manager.cc:1151] Deleting tablet 3b7d85a2941545ceaa16b22e8328b4ff
1026 02:56:33.593251 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.593322 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4627053}
I20251026 02:56:33.598047 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 441) took 4516 ms. Trace:
I20251026 02:56:33.600996 2375 tablet_bootstrap.cc:492] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.601156 2375 ts_tablet_manager.cc:1403] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.003s
I20251026 02:56:33.601614 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 58d60bf2a9e546e6a9474a5af6d5273a (DEFAULT_TABLE table=default.Table_cef54300186c47eca175dff1adf385db [id=6b5fd8a21a8f42718d77deb53f71f414]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.601969 2375 raft_consensus.cc:359] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.602164 2375 raft_consensus.cc:385] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.602206 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 3efafaac6c634d088396d56c8debd990 (DEFAULT_TABLE table=default.Table_44853e936f824114a26a5c01ae8543d3 [id=0558cab549f94833aed58ea430e03566]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.602232 2375 raft_consensus.cc:740] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.601969 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d60bf2a9e546e6a9474a5af6d5273a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.602617 2375 consensus_queue.cc:260] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.603116 2375 ts_tablet_manager.cc:1434] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:33.603188 2934 raft_consensus.cc:2749] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:33.603386 2375 tablet_bootstrap.cc:492] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.605202 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 1ffead7ca2d946c78aa4b8a602865fc8 (DEFAULT_TABLE table=default.Table_ac31a485be77410ca1fafabe643f37a2 [id=1a611c5db865445e840b4e227852e819]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.605511 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ffead7ca2d946c78aa4b8a602865fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.582990 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 493) took 2457 ms. Trace:
I20251026 02:56:33.605965 2372 rpcz_store.cc:276] 1026 02:56:31.125345 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.127047 (+ 1702us) service_pool.cc:225] Handling call
1026 02:56:33.297435 (+2170388us) ts_tablet_manager.cc:1151] Deleting tablet ca4611c0a494425ca50d599b310872c8
1026 02:56:33.297446 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.336449 (+ 39003us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.439575 (+103126us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.582969 (+143394us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2170211}
I20251026 02:56:33.606299 2372 tablet_replica.cc:333] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:33.606477 2372 raft_consensus.cc:2243] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:33.606640 2372 raft_consensus.cc:2272] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:33.607764 2372 ts_tablet_manager.cc:1916] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:33.609257 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efafaac6c634d088396d56c8debd990. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.609267 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet ca4611c0a494425ca50d599b310872c8 (table default.Table_21d71fb70373400396a75b6e52416078 [id=2dcd0d98a1354635a3673182e6fad58a]) successfully deleted
I20251026 02:56:33.611820 2377 rpcz_store.cc:276] 1026 02:56:29.081362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.081641 (+ 279us) service_pool.cc:225] Handling call
1026 02:56:33.597914 (+4516273us) ts_tablet_manager.cc:1151] Deleting tablet be1c0948c0ae4563b7cb644521914cb9
1026 02:56:33.597923 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.598003 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4510883}
I20251026 02:56:33.612061 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_11f1980aeb6a4f38a5c9eebc431ddd86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:33.612203 2377 tablet_replica.cc:333] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:33.612402 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet dbfe80d388644d0488cb2431a334c908 (table default.Table_ce7dea3ace314a5fa3ffeb9f16512725 [id=5964dceedb1c4d69b39051f9fe86b9dd]) successfully deleted
W20251026 02:56:33.612720 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet dfd3e23d83a84648bd11cd258377aecb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfd3e23d83a84648bd11cd258377aecb already in progress: creating tablet
W20251026 02:56:33.612988 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3b7d85a2941545ceaa16b22e8328b4ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b7d85a2941545ceaa16b22e8328b4ff already in progress: creating tablet
I20251026 02:56:33.613215 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 1ffead7ca2d946c78aa4b8a602865fc8 (DEFAULT_TABLE table=default.Table_ac31a485be77410ca1fafabe643f37a2 [id=1a611c5db865445e840b4e227852e819]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.613703 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ffead7ca2d946c78aa4b8a602865fc8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:33.613724 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11f1980aeb6a4f38a5c9eebc431ddd86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.613991 2375 tablet_bootstrap.cc:654] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:33.614820 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet be1c0948c0ae4563b7cb644521914cb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be1c0948c0ae4563b7cb644521914cb9 already in progress: creating tablet
I20251026 02:56:33.615727 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 58d60bf2a9e546e6a9474a5af6d5273a (DEFAULT_TABLE table=default.Table_cef54300186c47eca175dff1adf385db [id=6b5fd8a21a8f42718d77deb53f71f414]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.615955 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 1bed60c80d584cba812f045d94995098 (DEFAULT_TABLE table=default.Table_38566e9ac15c44b9a28b046e47bf9448 [id=5db2a0336ff94ff2bc37d42bf3d9d9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.616128 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d60bf2a9e546e6a9474a5af6d5273a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.616504 2230 tablet_service.cc:1505] Processing CreateTablet for tablet e044af9739364ecdbbaddc115c09bfab (DEFAULT_TABLE table=default.Table_6b7a7d49a4cf4c7084d664fa624c3da4 [id=5e5fa8c80fd24402a3e9bd94b263bff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.616840 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e044af9739364ecdbbaddc115c09bfab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.618880 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 1bed60c80d584cba812f045d94995098 (DEFAULT_TABLE table=default.Table_38566e9ac15c44b9a28b046e47bf9448 [id=5db2a0336ff94ff2bc37d42bf3d9d9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.619494 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 1bed60c80d584cba812f045d94995098 (DEFAULT_TABLE table=default.Table_38566e9ac15c44b9a28b046e47bf9448 [id=5db2a0336ff94ff2bc37d42bf3d9d9f8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:33.619885 2194 consensus_peers.cc:597] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98fe44aab33b40eba819386c3bffca46. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:33.620671 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 657e07d0b17c44d4b78b97210f17ce40 (DEFAULT_TABLE table=default.Table_76c2c5fa88984a46ac25402fa3d04654 [id=5ff1ef27629b4ba4950c0dc5ceb41d3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.621367 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 657e07d0b17c44d4b78b97210f17ce40 (DEFAULT_TABLE table=default.Table_76c2c5fa88984a46ac25402fa3d04654 [id=5ff1ef27629b4ba4950c0dc5ceb41d3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.621460 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 657e07d0b17c44d4b78b97210f17ce40 (DEFAULT_TABLE table=default.Table_76c2c5fa88984a46ac25402fa3d04654 [id=5ff1ef27629b4ba4950c0dc5ceb41d3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.622128 3028 raft_consensus.cc:493] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.622334 3028 raft_consensus.cc:515] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.623564 3028 leader_election.cc:290] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:33.625484 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352a284902c94f088e2e6ae006162db0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:33.627025 2194 leader_election.cc:343] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.628649 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 049a915b35224cdaa38df77c734a0f01
I20251026 02:56:33.629150 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352a284902c94f088e2e6ae006162db0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:33.629418 2118 raft_consensus.cc:2468] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:33.629930 3006 raft_consensus.cc:493] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.630071 3006 raft_consensus.cc:515] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.630741 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 049a915b35224cdaa38df77c734a0f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.630824 3006 leader_election.cc:290] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.631418 2195 leader_election.cc:304] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.631647 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6a273a7631fc410e8ba77d604a7e1c66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:33.632148 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 049a915b35224cdaa38df77c734a0f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.632165 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 049a915b35224cdaa38df77c734a0f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:33.632319 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a273a7631fc410e8ba77d604a7e1c66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.632877 3150 raft_consensus.cc:2804] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:33.632994 3150 raft_consensus.cc:493] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:33.633108 3150 raft_consensus.cc:3060] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.633558 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1571e787564342cd8a0f26f710f89b0c" } new_table_name: "default.Table_71a2681a68ef4299afc9e7d30389b870" modify_external_catalogs: true
I20251026 02:56:33.634747 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4996540a4e3749c6aa9bdecf10b2fae3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.635248 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4996540a4e3749c6aa9bdecf10b2fae3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.637326 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 82e4bc6195484074b0da610a8d1b65bc (DEFAULT_TABLE table=default.Table_6da5ba99f77e43cc93a15171e7355167 [id=72e0e50b64a6432a97a1f7c3edf0e286]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.638038 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_df797fe86da84688840a0e8a2d2f18ed" } new_table_name: "default.Table_217ad44ad25140129bf8e39c88b3b593" modify_external_catalogs: true
I20251026 02:56:33.638972 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efd751ee4b1d433198dcd00c081f648d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.639348 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efd751ee4b1d433198dcd00c081f648d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:33.640801 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 82e4bc6195484074b0da610a8d1b65bc (DEFAULT_TABLE table=default.Table_6da5ba99f77e43cc93a15171e7355167 [id=72e0e50b64a6432a97a1f7c3edf0e286]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.641032 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 82e4bc6195484074b0da610a8d1b65bc (DEFAULT_TABLE table=default.Table_6da5ba99f77e43cc93a15171e7355167 [id=72e0e50b64a6432a97a1f7c3edf0e286]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.641845 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 5e7c609238394fb7ab1597556894774a (DEFAULT_TABLE table=default.Table_3b00d3dd0fe54a4b867f4dcf830d98a9 [id=6d81323b0fa44961a93b448336db42d3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:33.642380 2053 leader_election.cc:343] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:33.642633 2053 leader_election.cc:343] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.642740 2053 leader_election.cc:304] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.643081 3006 raft_consensus.cc:2749] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:33.645991 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 046a34f9408b406ba08471421f422064 consensus state has no leader
I20251026 02:56:33.646443 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e250def75d2042ee89b9ce3bd7ed34ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.646961 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e250def75d2042ee89b9ce3bd7ed34ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.647368 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 5e7c609238394fb7ab1597556894774a (DEFAULT_TABLE table=default.Table_3b00d3dd0fe54a4b867f4dcf830d98a9 [id=6d81323b0fa44961a93b448336db42d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.648591 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet ed836bb7519e4b0290e6547fa57128bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.649160 2089 tablet_service.cc:1505] Processing CreateTablet for tablet 52a1312bf5d044b1b21454f6cefe7980 (DEFAULT_TABLE table=default.Table_f6a0eda989c8467cbb302a20f18c402e [id=e32439f487b84405b4aaec850942e275]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.650214 2375 tablet_bootstrap.cc:492] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.650341 2375 ts_tablet_manager.cc:1403] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.003s
I20251026 02:56:33.651322 2375 raft_consensus.cc:359] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.651482 2375 raft_consensus.cc:385] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.651587 2375 raft_consensus.cc:740] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.651816 2375 consensus_queue.cc:260] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.652395 2375 ts_tablet_manager.cc:1434] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:33.652556 2226 tablet_service.cc:1505] Processing CreateTablet for tablet 52a1312bf5d044b1b21454f6cefe7980 (DEFAULT_TABLE table=default.Table_f6a0eda989c8467cbb302a20f18c402e [id=e32439f487b84405b4aaec850942e275]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.652829 2375 tablet_bootstrap.cc:492] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.653100 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 52a1312bf5d044b1b21454f6cefe7980 (DEFAULT_TABLE table=default.Table_f6a0eda989c8467cbb302a20f18c402e [id=e32439f487b84405b4aaec850942e275]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.653843 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657e07d0b17c44d4b78b97210f17ce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.655453 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 9b0406ceb60b48969ca84f6268b43dbb (DEFAULT_TABLE table=default.Table_dd6ea2361b074225af91ade940fbf0d9 [id=b8cdc32c8dfe416b89bcf9b5fa5c2ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.655977 3166 raft_consensus.cc:493] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.656170 3166 raft_consensus.cc:515] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.656544 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bed60c80d584cba812f045d94995098. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.656845 3166 leader_election.cc:290] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.658274 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 5e7c609238394fb7ab1597556894774a (DEFAULT_TABLE table=default.Table_3b00d3dd0fe54a4b867f4dcf830d98a9 [id=6d81323b0fa44961a93b448336db42d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.658640 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e7c609238394fb7ab1597556894774a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.660899 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5226d4516ce24ecda51dab35c86da132" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:33.661978 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a1312bf5d044b1b21454f6cefe7980. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.662024 2375 tablet_bootstrap.cc:654] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.662046 2227 tablet_service.cc:1505] Processing CreateTablet for tablet 9b0406ceb60b48969ca84f6268b43dbb (DEFAULT_TABLE table=default.Table_dd6ea2361b074225af91ade940fbf0d9 [id=b8cdc32c8dfe416b89bcf9b5fa5c2ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.662401 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b0406ceb60b48969ca84f6268b43dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.663151 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5226d4516ce24ecda51dab35c86da132" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.663787 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 9b0406ceb60b48969ca84f6268b43dbb (DEFAULT_TABLE table=default.Table_dd6ea2361b074225af91ade940fbf0d9 [id=b8cdc32c8dfe416b89bcf9b5fa5c2ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.664016 1662 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_40e6c3d6d5364f56a9517c1bfe79f773 [id=c8435cef755d4de69c33f599df1c0f75] alter complete (version 2)
I20251026 02:56:33.664340 1915 leader_election.cc:304] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:33.665064 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 4a5b68772a66402bbb0b7e00bf96fcc0
I20251026 02:56:33.665194 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82e4bc6195484074b0da610a8d1b65bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.668267 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a5b68772a66402bbb0b7e00bf96fcc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.668563 3166 raft_consensus.cc:2749] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:33.670503 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a5b68772a66402bbb0b7e00bf96fcc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.670845 2377 raft_consensus.cc:2243] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:33.671253 2377 raft_consensus.cc:2272] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:33.671499 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f15000847df44f1ebebad3ca2c2b8046. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.671828 2377 ts_tablet_manager.cc:1916] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:33.672063 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cedff3287c7b4a07a81462e8c4b44970. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.672843 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a5b68772a66402bbb0b7e00bf96fcc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:33.674218 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8fe7825fa8d46669772cf84bc4b3afc consensus state has no leader
I20251026 02:56:33.675907 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed205c1800243ed9830d720ac85de2f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:33.676168 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b9a5daa36b547a7888d38dc3329eaf2 consensus state has no leader
I20251026 02:56:33.676522 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82e4bc6195484074b0da610a8d1b65bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.676899 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efafaac6c634d088396d56c8debd990. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.677290 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d60bf2a9e546e6a9474a5af6d5273a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.677663 1949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ffead7ca2d946c78aa4b8a602865fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.678097 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b0406ceb60b48969ca84f6268b43dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.678412 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e7c609238394fb7ab1597556894774a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.681715 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e044af9739364ecdbbaddc115c09bfab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.684397 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4005a095c594469b8d8105a0679ff45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.687335 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 5aff38ba32864ad18b5d808e5235efe3 (DEFAULT_TABLE table=default.Table_98bf1a1deafd476e85a4d04987d9355d [id=6b3ebeed67e04a0088206b95c2a875bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.688496 3164 raft_consensus.cc:515] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.689344 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 144b957c69e64085a6f78050dea426b4 (DEFAULT_TABLE table=default.Table_bc49906123d84d55a0849db64bb7c25a [id=280fd9f13d9c41f682830f9475a54a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.689852 3164 leader_election.cc:290] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.691460 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 46547883cfe041f3954aa4128fba1d8c (DEFAULT_TABLE table=default.Table_7ece4924086a4ae68db59650193f0e0e [id=95950afc203c4846944d24c3bf1608cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.693785 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 31edacb952514c8aa25c79f509869e8f (DEFAULT_TABLE table=default.Table_103c3d38235f4771a4cad7b79143a809 [id=3ea80abaf10d48b79afcd263ad9c1c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.695933 2094 tablet_service.cc:1505] Processing CreateTablet for tablet c2ca394b096f4fd88f8503488b08d8d5 (DEFAULT_TABLE table=default.Table_de1a357fbcb3425da0335f0ef9772de9 [id=aabd61cb9b2a4a5d9455bfd11340704d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:33.697026 1916 leader_election.cc:343] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: b1a2522801364693be0bda6fa45a697e
I20251026 02:56:33.697708 3144 raft_consensus.cc:493] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.697829 3144 raft_consensus.cc:515] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.698464 3144 leader_election.cc:290] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:33.699326 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "276bd3b97c97426daf802e1f57feb265" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:33.700955 1981 raft_consensus.cc:3060] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:33.701545 2194 leader_election.cc:343] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.702247 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1a2522801364693be0bda6fa45a697e" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:33.702431 2260 raft_consensus.cc:3060] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.703138 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "276bd3b97c97426daf802e1f57feb265" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:33.703358 2118 raft_consensus.cc:2468] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:33.703864 3150 raft_consensus.cc:515] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.704814 2962 raft_consensus.cc:493] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.704926 2962 raft_consensus.cc:515] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.705444 2114 raft_consensus.cc:2468] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:33.706451 1979 raft_consensus.cc:3060] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.710305 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657e07d0b17c44d4b78b97210f17ce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.713745 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a1312bf5d044b1b21454f6cefe7980. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.716362 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bed60c80d584cba812f045d94995098. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:33.717715 3190 tablet_replica.cc:1410] Time spent flushing the Log queue took a long time: real 0.305s user 0.000s sys 0.000s
I20251026 02:56:33.723591 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_dd6ea2361b074225af91ade940fbf0d9" } modify_external_catalogs: true
I20251026 02:56:33.723824 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_dd6ea2361b074225af91ade940fbf0d9" } modify_external_catalogs: true
I20251026 02:56:33.723865 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5322450167b9484c9e1448d711dc9702"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.724591 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5322450167b9484c9e1448d711dc9702 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.725178 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_71583d73a49142ae9fec31193fb53542" } modify_external_catalogs: true
I20251026 02:56:33.725328 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_71583d73a49142ae9fec31193fb53542" } modify_external_catalogs: true
I20251026 02:56:33.727535 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d450815506d645aab06dee257de7ea3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.728087 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d450815506d645aab06dee257de7ea3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.731225 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 144b957c69e64085a6f78050dea426b4 (DEFAULT_TABLE table=default.Table_bc49906123d84d55a0849db64bb7c25a [id=280fd9f13d9c41f682830f9475a54a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.731487 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 5aff38ba32864ad18b5d808e5235efe3 (DEFAULT_TABLE table=default.Table_98bf1a1deafd476e85a4d04987d9355d [id=6b3ebeed67e04a0088206b95c2a875bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.731691 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 144b957c69e64085a6f78050dea426b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.731984 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 31edacb952514c8aa25c79f509869e8f (DEFAULT_TABLE table=default.Table_103c3d38235f4771a4cad7b79143a809 [id=3ea80abaf10d48b79afcd263ad9c1c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.732342 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31edacb952514c8aa25c79f509869e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.733479 2375 tablet_bootstrap.cc:492] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.733652 2375 ts_tablet_manager.cc:1403] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.002s
I20251026 02:56:33.733865 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 46547883cfe041f3954aa4128fba1d8c (DEFAULT_TABLE table=default.Table_7ece4924086a4ae68db59650193f0e0e [id=95950afc203c4846944d24c3bf1608cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.734445 2375 raft_consensus.cc:359] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.734654 2375 raft_consensus.cc:385] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.734736 2375 raft_consensus.cc:740] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.734982 2375 consensus_queue.cc:260] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.735181 3006 raft_consensus.cc:493] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.735378 3006 raft_consensus.cc:515] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.735605 2375 ts_tablet_manager.cc:1434] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:33.735925 2375 tablet_bootstrap.cc:492] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.735986 3006 leader_election.cc:290] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.736951 1947 tablet_service.cc:1505] Processing CreateTablet for tablet c2ca394b096f4fd88f8503488b08d8d5 (DEFAULT_TABLE table=default.Table_de1a357fbcb3425da0335f0ef9772de9 [id=aabd61cb9b2a4a5d9455bfd11340704d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.737370 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aff38ba32864ad18b5d808e5235efe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.738888 1662 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa] alter complete (version 1)
I20251026 02:56:33.739071 2372 ts_tablet_manager.cc:1929] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:33.739223 2375 tablet_bootstrap.cc:654] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.739244 2372 log.cc:1199] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/22e4a433801a4cb58062d3e76c760b69
I20251026 02:56:33.740677 2372 ts_tablet_manager.cc:1950] T 22e4a433801a4cb58062d3e76c760b69 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:33.742239 2098 tablet_service.cc:1505] Processing CreateTablet for tablet c09153fbfe18434393c188058a3461d5 (DEFAULT_TABLE table=default.Table_908a8748249c46ad964acc6abf54cd58 [id=bbf21d92a3db4b1a837deb2f5f3fe3cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.747189 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 59f5078f03224ef18ef4b6864be7122b (DEFAULT_TABLE table=default.Table_f28a7b90302c4e3993516e647d2bc67c [id=49086aec5a2b49bbb54e04f181f6c7fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.755786 3150 leader_election.cc:290] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:33.704383 2195 leader_election.cc:304] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:33.756485 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 52124.9us
I20251026 02:56:33.756464 2093 tablet_service.cc:1505] Processing CreateTablet for tablet d9391e7654c442a5bb312f0daaf1d99b (DEFAULT_TABLE table=default.Table_2e3bf8087ce7480584b9b66c8270c451 [id=0f4a4be6bea7411aa2e13984864e827c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.756883 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9391e7654c442a5bb312f0daaf1d99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.756906 3150 raft_consensus.cc:2804] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:33.756982 3150 raft_consensus.cc:493] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:33.757061 3150 raft_consensus.cc:3060] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.757119 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82e4bc6195484074b0da610a8d1b65bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.757289 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bed60c80d584cba812f045d94995098. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.757606 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f15000847df44f1ebebad3ca2c2b8046. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.757786 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aff38ba32864ad18b5d808e5235efe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.758080 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4005a095c594469b8d8105a0679ff45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.758282 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e7c609238394fb7ab1597556894774a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.758702 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2ca394b096f4fd88f8503488b08d8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.759186 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31edacb952514c8aa25c79f509869e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.759548 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b0406ceb60b48969ca84f6268b43dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.759943 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 144b957c69e64085a6f78050dea426b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.760347 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657e07d0b17c44d4b78b97210f17ce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.760684 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46547883cfe041f3954aa4128fba1d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.761015 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a1312bf5d044b1b21454f6cefe7980. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.761888 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c09153fbfe18434393c188058a3461d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.763561 2962 leader_election.cc:290] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:33.765291 3214 tablet.cc:1722] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:33.770936 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352a284902c94f088e2e6ae006162db0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:33.771163 2114 raft_consensus.cc:3060] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.772514 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59f5078f03224ef18ef4b6864be7122b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.777921 2084 tablet_service.cc:1505] Processing CreateTablet for tablet a29ccd3f7b204cccb8c82d1589299350 (DEFAULT_TABLE table=default.Table_5b95c2f1040441baa94ad346e22b4c72 [id=c82e2aa4eced4a8da31014e4b768ab2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.778604 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 5caeb11322fe48348ebb523967152187 with delete_type TABLET_DATA_DELETED (replaced by tablet d43b6f22dd4f4f9c9657f21514550312) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.779088 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 59f5078f03224ef18ef4b6864be7122b (DEFAULT_TABLE table=default.Table_f28a7b90302c4e3993516e647d2bc67c [id=49086aec5a2b49bbb54e04f181f6c7fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.779145 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 46547883cfe041f3954aa4128fba1d8c (DEFAULT_TABLE table=default.Table_7ece4924086a4ae68db59650193f0e0e [id=95950afc203c4846944d24c3bf1608cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.779461 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59f5078f03224ef18ef4b6864be7122b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.779654 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 144b957c69e64085a6f78050dea426b4 (DEFAULT_TABLE table=default.Table_bc49906123d84d55a0849db64bb7c25a [id=280fd9f13d9c41f682830f9475a54a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.779987 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet 0fe0e06a86e241bc91a1db4219fe25bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.775417 2240 tablet_service.cc:1505] Processing CreateTablet for tablet c09153fbfe18434393c188058a3461d5 (DEFAULT_TABLE table=default.Table_908a8748249c46ad964acc6abf54cd58 [id=bbf21d92a3db4b1a837deb2f5f3fe3cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.780287 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46547883cfe041f3954aa4128fba1d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.781400 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 5d7261ee6f244e77891e37a4c1a984ed (DEFAULT_TABLE table=default.Table_d1d2850518cc4c51a5cd3296d2d19680 [id=e6b5e81aa90d44a79a7994bb4d4877ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.782394 3144 raft_consensus.cc:493] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.784013 1662 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 8f3ae3a377af46449c3faca339f86053 (table default.Table_6a9abf3b541f42cda3bbf2f44d86c970 [id=31f977e55ba041f59bcc721d469746fd]). Expected version 1 got 0
I20251026 02:56:33.783927 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 5aff38ba32864ad18b5d808e5235efe3 (DEFAULT_TABLE table=default.Table_98bf1a1deafd476e85a4d04987d9355d [id=6b3ebeed67e04a0088206b95c2a875bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.784590 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c9c7b9787c3a458ea41be132b0841599" } new_table_name: "default.Table_76e2e866c5b84956aac1e1bcde61692e" modify_external_catalogs: true
I20251026 02:56:33.786769 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 358e805370ef410e9d65ee92a8d04f34
I20251026 02:56:33.787910 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9b0406ceb60b48969ca84f6268b43dbb
I20251026 02:56:33.789127 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d52e76dbc27d4816bb3346700e4b4e8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:33.789474 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 358e805370ef410e9d65ee92a8d04f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:33.789745 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d52e76dbc27d4816bb3346700e4b4e8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.789765 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b0406ceb60b48969ca84f6268b43dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.791020 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa0aa6cdd0de4b0a91f553c2341e9c01" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.791148 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46547883cfe041f3954aa4128fba1d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.791525 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3050e72d42417ab58a05f7e3aa56b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.791954 2376 ts_tablet_manager.cc:1929] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:33.791954 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d48e174e391f42d6aa0b05c322b5b828" } modify_external_catalogs: true
I20251026 02:56:33.792109 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d48e174e391f42d6aa0b05c322b5b828" } modify_external_catalogs: true
I20251026 02:56:33.793427 3144 raft_consensus.cc:515] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.779106 2195 leader_election.cc:304] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.795641 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 45e79be9b98945c49fb712019b21b91b (DEFAULT_TABLE table=default.Table_c985fc44cad2462f8e82eadf887551c8 [id=09dc650467064e92a056f6344fda1c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.795712 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 31edacb952514c8aa25c79f509869e8f (DEFAULT_TABLE table=default.Table_103c3d38235f4771a4cad7b79143a809 [id=3ea80abaf10d48b79afcd263ad9c1c36]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:33.796447 2053 leader_election.cc:343] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.796808 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5caeb11322fe48348ebb523967152187" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:33.797020 2118 raft_consensus.cc:2468] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:33.797750 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa0aa6cdd0de4b0a91f553c2341e9c01" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:33.790628 2224 tablet_service.cc:1505] Processing CreateTablet for tablet d9391e7654c442a5bb312f0daaf1d99b (DEFAULT_TABLE table=default.Table_2e3bf8087ce7480584b9b66c8270c451 [id=0f4a4be6bea7411aa2e13984864e827c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.798385 2243 tablet_service.cc:1505] Processing CreateTablet for tablet c2ca394b096f4fd88f8503488b08d8d5 (DEFAULT_TABLE table=default.Table_de1a357fbcb3425da0335f0ef9772de9 [id=aabd61cb9b2a4a5d9455bfd11340704d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.798629 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 2524ee0c964345459c3e62f1e39b8f9e with delete_type TABLET_DATA_DELETED (replaced by tablet 5a00bc04b3654f90af342986751344e6) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.798903 2237 tablet_service.cc:1505] Processing CreateTablet for tablet a29ccd3f7b204cccb8c82d1589299350 (DEFAULT_TABLE table=default.Table_5b95c2f1040441baa94ad346e22b4c72 [id=c82e2aa4eced4a8da31014e4b768ab2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.800971 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 4b396cec093c4f40844b901550eee1f5 (DEFAULT_TABLE table=default.Table_8929007b52484dcca450eb9ea54fe75d [id=6ce21ddef8314008a1e55705ed6c1358]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.802757 3144 leader_election.cc:290] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:33.805378 3164 raft_consensus.cc:493] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.806900 2381 tablet_bootstrap.cc:492] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:33.807032 2381 ts_tablet_manager.cc:1403] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.412s user 0.006s sys 0.013s
I20251026 02:56:33.813490 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7062c5ff91e14369ab69a2d199585c55
I20251026 02:56:33.815174 2375 tablet_bootstrap.cc:492] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.815399 2375 ts_tablet_manager.cc:1403] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.006s
I20251026 02:56:33.816299 2375 raft_consensus.cc:359] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.816560 2375 raft_consensus.cc:385] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.816699 2375 raft_consensus.cc:740] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.817000 2375 consensus_queue.cc:260] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.817546 2375 ts_tablet_manager.cc:1434] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:33.818027 2375 tablet_bootstrap.cc:492] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.819795 2375 tablet_bootstrap.cc:654] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.826682 3065 raft_consensus.cc:2804] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:33.826910 3065 raft_consensus.cc:697] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:33.827378 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 488) took 2702 ms. Trace:
I20251026 02:56:33.827526 2372 rpcz_store.cc:276] 1026 02:56:31.124623 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.128754 (+ 4131us) service_pool.cc:225] Handling call
1026 02:56:33.606261 (+2477507us) ts_tablet_manager.cc:1151] Deleting tablet 22e4a433801a4cb58062d3e76c760b69
1026 02:56:33.606270 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.676447 (+ 70177us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.738884 (+ 62437us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.827332 (+ 88448us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2477330}
I20251026 02:56:33.827736 1951 tablet_service.cc:1505] Processing CreateTablet for tablet c09153fbfe18434393c188058a3461d5 (DEFAULT_TABLE table=default.Table_908a8748249c46ad964acc6abf54cd58 [id=bbf21d92a3db4b1a837deb2f5f3fe3cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.827920 2372 tablet_replica.cc:333] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:33.828123 2372 raft_consensus.cc:2243] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:33.828333 2372 raft_consensus.cc:2272] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:33.828966 2372 ts_tablet_manager.cc:1916] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:33.829246 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2ca394b096f4fd88f8503488b08d8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.829774 3150 raft_consensus.cc:515] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.830093 3164 raft_consensus.cc:515] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.830749 3164 leader_election.cc:290] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.831554 2114 raft_consensus.cc:2468] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:33.831678 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352a284902c94f088e2e6ae006162db0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:33.832281 1916 leader_election.cc:343] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ca4611c0a494425ca50d599b310872c8
I20251026 02:56:33.832739 2376 log.cc:1199] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/c570e5f9a93c45bda32ec92c68794714
I20251026 02:56:33.834976 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c09153fbfe18434393c188058a3461d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.835652 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_36597a9424aa42128d73327bd78be6fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.836318 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36597a9424aa42128d73327bd78be6fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.840301 3163 raft_consensus.cc:515] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.841928 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 22e4a433801a4cb58062d3e76c760b69 (table default.Table_f7c216f3ebd34c718ceef042a435b4cd [id=b4c0611288a949378eed16d32b5170b9]) successfully deleted
I20251026 02:56:33.830430 3150 leader_election.cc:290] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:33.843266 3163 leader_election.cc:290] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.843699 2376 ts_tablet_manager.cc:1950] T c570e5f9a93c45bda32ec92c68794714 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:33.843675 2086 tablet_service.cc:1505] Processing CreateTablet for tablet a956ebfb1eaa4d6fb1575b4165293193 (DEFAULT_TABLE table=default.Table_9b0942af08414e7bb088a14c781f0edd [id=5fb61e88f7074976bac29b40f4466ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.843878 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5caeb11322fe48348ebb523967152187" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:33.844059 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a956ebfb1eaa4d6fb1575b4165293193. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:33.852571 1916 leader_election.cc:343] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: a70d70abc3a34dac8cf9a747903fa78c
I20251026 02:56:33.852582 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 1fba4c45dc0448fc95469df5be374809 (DEFAULT_TABLE table=default.Table_472acbcc3a244422abe7a7477c65c87a [id=fb698813057b43cd940cadac51367204]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.852995 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fba4c45dc0448fc95469df5be374809. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.853207 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2524ee0c964345459c3e62f1e39b8f9e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:33.855449 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 7062c5ff91e14369ab69a2d199585c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.857213 1977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "276bd3b97c97426daf802e1f57feb265" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:33.860072 3006 raft_consensus.cc:493] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.860237 3006 raft_consensus.cc:515] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.860819 3006 leader_election.cc:290] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.862247 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29ccd3f7b204cccb8c82d1589299350. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.862758 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d7261ee6f244e77891e37a4c1a984ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.863132 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45e79be9b98945c49fb712019b21b91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.863585 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b396cec093c4f40844b901550eee1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.864634 1977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c3050e72d42417ab58a05f7e3aa56b3" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:33.865167 2194 leader_election.cc:343] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:33.865497 2194 leader_election.cc:343] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:33.866149 2194 leader_election.cc:343] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:33.866314 2194 consensus_peers.cc:597] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22e4a433801a4cb58062d3e76c760b69. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:33.866443 2194 leader_election.cc:343] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:33.867256 2053 leader_election.cc:343] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.867401 2053 leader_election.cc:304] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:33.867950 2053 leader_election.cc:343] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.868156 3006 raft_consensus.cc:2749] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:33.869825 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7667f4f99bf744579f1dc2b2d0f469eb" } modify_external_catalogs: true
I20251026 02:56:33.870049 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7667f4f99bf744579f1dc2b2d0f469eb" } modify_external_catalogs: true
I20251026 02:56:33.870610 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b51c545ae1d248b293e87daf463aaf8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.871191 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b51c545ae1d248b293e87daf463aaf8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.871488 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2704c1510c6442c1b922a5ed64bb7084"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.871937 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2704c1510c6442c1b922a5ed64bb7084 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.872437 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6da5ba99f77e43cc93a15171e7355167" } modify_external_catalogs: true
I20251026 02:56:33.872599 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6da5ba99f77e43cc93a15171e7355167" } modify_external_catalogs: true
I20251026 02:56:33.873111 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_afa5656eb2a74f6ca79f85270f7c97eb" } new_table_name: "default.Table_6798c68a24044bce9d44d8f21417ebf0" modify_external_catalogs: true
I20251026 02:56:33.874274 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_559b75aad2cb44b196cc0ab1dbf2c550" } new_table_name: "default.Table_9fef71477b964049a34fada596af80bf" modify_external_catalogs: true
I20251026 02:56:33.877671 2378 tablet_bootstrap.cc:492] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:33.877830 2378 ts_tablet_manager.cc:1403] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.444s user 0.040s sys 0.057s
I20251026 02:56:33.877909 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b5ccd4d8815f49b192cb9f1973042cce" } new_table_name: "default.Table_455479b9f17040e48d4cd186bacef556" modify_external_catalogs: true
I20251026 02:56:33.878661 2378 raft_consensus.cc:359] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.878822 2378 raft_consensus.cc:385] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.878893 2378 raft_consensus.cc:740] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.879163 2378 consensus_queue.cc:260] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.880585 2378 ts_tablet_manager.cc:1434] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:33.880930 2378 tablet_bootstrap.cc:492] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:33.881379 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c25e8ab335f34833a18f68c6585fd7a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.881902 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c25e8ab335f34833a18f68c6585fd7a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.827204 3065 consensus_queue.cc:237] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.888446 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 5d7261ee6f244e77891e37a4c1a984ed (DEFAULT_TABLE table=default.Table_d1d2850518cc4c51a5cd3296d2d19680 [id=e6b5e81aa90d44a79a7994bb4d4877ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.889205 2378 tablet_bootstrap.cc:654] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.890271 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet d212170f056a44dfb15cfcb7f9fbae2f
I20251026 02:56:33.890458 2377 ts_tablet_manager.cc:1929] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:33.890584 2377 log.cc:1199] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/9ec035b1caac450d9ab55e74a061ae38
I20251026 02:56:33.890964 2377 ts_tablet_manager.cc:1950] T 9ec035b1caac450d9ab55e74a061ae38 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:33.896356 2381 raft_consensus.cc:359] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.896528 2381 raft_consensus.cc:385] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.896576 2381 raft_consensus.cc:740] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.896733 2381 consensus_queue.cc:260] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.897131 2381 ts_tablet_manager.cc:1434] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.090s user 0.002s sys 0.000s
I20251026 02:56:33.899746 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 358e805370ef410e9d65ee92a8d04f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.900252 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 45e79be9b98945c49fb712019b21b91b (DEFAULT_TABLE table=default.Table_c985fc44cad2462f8e82eadf887551c8 [id=09dc650467064e92a056f6344fda1c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.901142 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b0406ceb60b48969ca84f6268b43dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.901481 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "276bd3b97c97426daf802e1f57feb265" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:33.901516 2238 tablet_service.cc:1505] Processing CreateTablet for tablet 4b396cec093c4f40844b901550eee1f5 (DEFAULT_TABLE table=default.Table_8929007b52484dcca450eb9ea54fe75d [id=6ce21ddef8314008a1e55705ed6c1358]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:33.901652 2117 raft_consensus.cc:3060] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.902261 2195 leader_election.cc:304] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.902503 3028 raft_consensus.cc:2804] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:33.902565 3028 raft_consensus.cc:493] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:33.902616 3028 raft_consensus.cc:3060] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.903218 2962 consensus_queue.cc:1048] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:33.903805 2195 leader_election.cc:304] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.903983 2962 raft_consensus.cc:2804] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:33.904079 2962 raft_consensus.cc:697] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:33.904362 2962 consensus_queue.cc:237] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.904688 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2524ee0c964345459c3e62f1e39b8f9e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:33.904892 2114 raft_consensus.cc:2468] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:33.905545 2381 tablet_bootstrap.cc:492] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:33.905851 2258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a70d70abc3a34dac8cf9a747903fa78c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:33.906019 2258 raft_consensus.cc:3060] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.915023 3007 raft_consensus.cc:493] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.915189 3007 raft_consensus.cc:515] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.915768 3007 leader_election.cc:290] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.919638 1632 master_runner.cc:186] Error getting master registration for 127.1.150.60:33729: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111),
I20251026 02:56:33.919890 1632 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251026 02:56:33.921610 1915 leader_election.cc:343] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: ca4611c0a494425ca50d599b310872c8
I20251026 02:56:33.921803 1915 leader_election.cc:304] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:33.922905 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 82e4bc6195484074b0da610a8d1b65bc
W20251026 02:56:33.923653 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7a5020acff04c1d8b5749cdfd7dddfd has no consensus state
I20251026 02:56:33.923635 1977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce922040298d4952af13652eeda7a234" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:33.925199 2053 leader_election.cc:343] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.925292 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet d212170f056a44dfb15cfcb7f9fbae2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.925709 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 82e4bc6195484074b0da610a8d1b65bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:33.926852 3163 raft_consensus.cc:2749] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:33.927914 2381 tablet_bootstrap.cc:654] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.929315 2195 leader_election.cc:304] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:33.929536 3144 raft_consensus.cc:2804] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:33.929605 3144 raft_consensus.cc:493] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:33.929661 3144 raft_consensus.cc:3060] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:33.932021 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c09153fbfe18434393c188058a3461d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.932448 3190 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:33.932896 2260 raft_consensus.cc:2468] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:33.933851 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aff38ba32864ad18b5d808e5235efe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.934311 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 144b957c69e64085a6f78050dea426b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.934799 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d7261ee6f244e77891e37a4c1a984ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.935186 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b396cec093c4f40844b901550eee1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.935613 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45e79be9b98945c49fb712019b21b91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.935681 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce922040298d4952af13652eeda7a234" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:33.940836 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_672f95f49551403a99355e82b17eaece" } new_table_name: "default.Table_72ee520a0e4d45fb920d1e3cc7a426b3" modify_external_catalogs: true
I20251026 02:56:33.941365 2375 tablet_bootstrap.cc:492] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.942409 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7921be016a944a41953b751ba927dacd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.943117 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7921be016a944a41953b751ba927dacd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.944427 2375 ts_tablet_manager.cc:1403] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.126s user 0.001s sys 0.006s
W20251026 02:56:33.945434 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85420a102dae4269811dbdc64aebb20a consensus state has no leader
W20251026 02:56:33.946436 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 consensus state has no leader
I20251026 02:56:33.947645 2375 raft_consensus.cc:359] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.947808 2375 raft_consensus.cc:385] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.947860 2375 raft_consensus.cc:740] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.948071 2375 consensus_queue.cc:260] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.954227 2375 ts_tablet_manager.cc:1434] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251026 02:56:33.954543 2375 tablet_bootstrap.cc:492] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.956130 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_18ed9c54e04542528f377fd10c75cf9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.956799 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18ed9c54e04542528f377fd10c75cf9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.959079 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b906304b7fc9457f98068c196bdea574"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:33.959372 3007 raft_consensus.cc:493] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.959488 3007 raft_consensus.cc:515] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.960088 3007 leader_election.cc:290] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.960645 1977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44af74033ccf4d048475f12d164938f5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:33.960876 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b906304b7fc9457f98068c196bdea574 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:33.961122 2053 leader_election.cc:343] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:33.963528 2375 tablet_bootstrap.cc:654] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:33.967175 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0cf76ef8f6af445eb0ce15c8d79136c5 consensus state has no leader
I20251026 02:56:33.967239 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2ca394b096f4fd88f8503488b08d8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.967762 2117 raft_consensus.cc:2468] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:33.969054 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29ccd3f7b204cccb8c82d1589299350. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.969306 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31edacb952514c8aa25c79f509869e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.972456 3166 raft_consensus.cc:493] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:33.972635 3166 raft_consensus.cc:515] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.973218 3166 leader_election.cc:290] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:33.935993 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c3050e72d42417ab58a05f7e3aa56b3" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:33.976392 1916 leader_election.cc:343] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: e62064fbdffb4954a3a1227efd6327ee
I20251026 02:56:33.977154 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9391e7654c442a5bb312f0daaf1d99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:33.977686 2378 tablet_bootstrap.cc:492] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:33.977845 2378 ts_tablet_manager.cc:1403] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.097s user 0.001s sys 0.002s
I20251026 02:56:33.978600 2378 raft_consensus.cc:359] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.978904 2378 raft_consensus.cc:385] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.978968 2378 raft_consensus.cc:740] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.979226 2378 consensus_queue.cc:260] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:33.979713 2378 ts_tablet_manager.cc:1434] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 02:56:33.980862 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet d212170f056a44dfb15cfcb7f9fbae2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:33.988529 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_cf80bd3545d84f44bceb3b4e7cd5961b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:33.989202 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf80bd3545d84f44bceb3b4e7cd5961b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.991506 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_999c4a3cf18e418eaf40b7dbee0e5095" } modify_external_catalogs: true
I20251026 02:56:33.991681 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_999c4a3cf18e418eaf40b7dbee0e5095" } modify_external_catalogs: true
I20251026 02:56:33.992468 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_067a40b223504137ad333312ad69073a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:33.992885 2375 tablet_bootstrap.cc:492] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:33.993012 2375 ts_tablet_manager.cc:1403] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
W20251026 02:56:33.992997 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_067a40b223504137ad333312ad69073a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:33.993772 2375 raft_consensus.cc:359] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.993925 2375 raft_consensus.cc:385] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:33.993971 2375 raft_consensus.cc:740] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:33.994203 2375 consensus_queue.cc:260] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:33.994594 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3107109fa3984828a1e2651e63ffb1ed" } modify_external_catalogs: true
I20251026 02:56:33.994740 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3107109fa3984828a1e2651e63ffb1ed" } modify_external_catalogs: true
I20251026 02:56:33.995462 2375 ts_tablet_manager.cc:1434] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:33.995776 2375 tablet_bootstrap.cc:492] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:33.998220 2375 tablet_bootstrap.cc:654] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:33.999071 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_40e6c3d6d5364f56a9517c1bfe79f773" } new_table_name: "default.Table_5221808b553f44bc82646e59a4f6881e" modify_external_catalogs: true
I20251026 02:56:34.004473 2378 tablet_bootstrap.cc:492] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:34.006603 2378 tablet_bootstrap.cc:654] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:34.007123 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0198232bf3d14a08b9ef2331469c256c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:34.007768 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0198232bf3d14a08b9ef2331469c256c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.015476 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3792bcbbc16b4be0b113cb5f07b9261b" } modify_external_catalogs: true
I20251026 02:56:34.015769 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3792bcbbc16b4be0b113cb5f07b9261b" } modify_external_catalogs: true
I20251026 02:56:34.022797 2233 tablet_service.cc:1505] Processing CreateTablet for tablet a956ebfb1eaa4d6fb1575b4165293193 (DEFAULT_TABLE table=default.Table_9b0942af08414e7bb088a14c781f0edd [id=5fb61e88f7074976bac29b40f4466ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.023227 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a956ebfb1eaa4d6fb1575b4165293193. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.024880 2375 tablet_bootstrap.cc:492] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:34.025112 2375 ts_tablet_manager.cc:1403] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20251026 02:56:34.026288 2375 raft_consensus.cc:359] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.026438 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 442) took 4873 ms. Trace:
I20251026 02:56:34.026613 2375 raft_consensus.cc:385] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:34.026789 2375 raft_consensus.cc:740] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:34.027148 2375 consensus_queue.cc:260] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.027901 2375 ts_tablet_manager.cc:1434] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 02:56:34.028360 2375 tablet_bootstrap.cc:492] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:34.031231 2375 tablet_bootstrap.cc:654] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:34.036384 2375 tablet_bootstrap.cc:492] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:34.036626 2375 ts_tablet_manager.cc:1403] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20251026 02:56:34.038787 2375 raft_consensus.cc:359] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.039088 2375 raft_consensus.cc:385] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:34.039175 2375 raft_consensus.cc:740] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:34.039405 2375 consensus_queue.cc:260] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.040040 2375 ts_tablet_manager.cc:1434] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:34.040352 2375 tablet_bootstrap.cc:492] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:34.041992 2375 tablet_bootstrap.cc:654] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:34.049129 2378 tablet_bootstrap.cc:492] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:34.049301 2378 ts_tablet_manager.cc:1403] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.003s
I20251026 02:56:34.050112 2378 raft_consensus.cc:359] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.050305 2378 raft_consensus.cc:385] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:34.050518 2378 raft_consensus.cc:740] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:34.050783 2378 consensus_queue.cc:260] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.052721 2378 ts_tablet_manager.cc:1434] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251026 02:56:34.053961 2378 tablet_bootstrap.cc:492] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:34.054777 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 45e79be9b98945c49fb712019b21b91b (DEFAULT_TABLE table=default.Table_c985fc44cad2462f8e82eadf887551c8 [id=09dc650467064e92a056f6344fda1c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.055190 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45e79be9b98945c49fb712019b21b91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.055723 2378 tablet_bootstrap.cc:654] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:34.073094 3007 raft_consensus.cc:493] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.073335 3007 raft_consensus.cc:515] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.079619 3007 leader_election.cc:290] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:34.082015 2053 consensus_peers.cc:597] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:34.084756 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8688a0bf946841deb5d88c931538b129" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:34.085516 2053 leader_election.cc:343] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:34.026530 2377 rpcz_store.cc:276] 1026 02:56:29.152985 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.154038 (+ 1053us) service_pool.cc:225] Handling call
1026 02:56:33.612132 (+4458094us) ts_tablet_manager.cc:1151] Deleting tablet 9ec035b1caac450d9ab55e74a061ae38
1026 02:56:33.612142 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.675002 (+ 62860us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.890314 (+215312us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:34.026412 (+136098us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4414369}
I20251026 02:56:34.095223 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 447) took 4850 ms. Trace:
I20251026 02:56:34.095330 2377 rpcz_store.cc:276] 1026 02:56:29.244621 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.244695 (+ 74us) service_pool.cc:225] Handling call
1026 02:56:34.095047 (+4850352us) ts_tablet_manager.cc:1151] Deleting tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94
1026 02:56:34.095059 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:34.095193 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4844920}
I20251026 02:56:34.107089 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 7062c5ff91e14369ab69a2d199585c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:34.095726 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 451) took 4834 ms. Trace:
I20251026 02:56:34.107591 2377 rpcz_store.cc:276] 1026 02:56:29.261235 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.261296 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:34.095601 (+4834305us) ts_tablet_manager.cc:1151] Deleting tablet dbf2a1ce78154415b23ab6d0efbc1f52
1026 02:56:34.095614 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:34.095708 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4834004}
I20251026 02:56:34.108163 3250 tablet.cc:1722] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:34.113019 3166 raft_consensus.cc:493] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.113235 3166 raft_consensus.cc:515] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:34.114126 3166 leader_election.cc:290] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:34.125933 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1370b9ad917344c093e5da3d0e9b1cf0" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:34.129436 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 1fba4c45dc0448fc95469df5be374809 (DEFAULT_TABLE table=default.Table_472acbcc3a244422abe7a7477c65c87a [id=fb698813057b43cd940cadac51367204]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.130321 2377 tablet_replica.cc:333] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:34.130522 2377 raft_consensus.cc:2243] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:34.130722 2377 raft_consensus.cc:2272] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:34.145103 2372 ts_tablet_manager.cc:1929] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:34.145294 2372 log.cc:1199] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/a111848706b743c9beb2fbd21f59d7f3
I20251026 02:56:34.149304 2375 tablet_bootstrap.cc:492] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:34.149631 2375 ts_tablet_manager.cc:1403] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.109s user 0.007s sys 0.015s
I20251026 02:56:34.150605 2375 raft_consensus.cc:359] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.150892 2375 raft_consensus.cc:385] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:34.151209 2375 raft_consensus.cc:740] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:34.151580 2375 consensus_queue.cc:260] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.160086 2375 ts_tablet_manager.cc:1434] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251026 02:56:34.160648 2375 tablet_bootstrap.cc:492] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:34.162854 2375 tablet_bootstrap.cc:654] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:34.167812 3007 raft_consensus.cc:493] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.167999 3007 raft_consensus.cc:515] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.168924 3007 leader_election.cc:290] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:34.170331 2377 ts_tablet_manager.cc:1916] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:34.173794 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dba102838945453cbf44e1cfe505a52c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:34.174715 2053 leader_election.cc:343] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:34.191232 3163 raft_consensus.cc:493] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 696476265495492790739ec126a08539)
I20251026 02:56:34.191426 3163 raft_consensus.cc:515] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:34.192261 3163 leader_election.cc:290] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:34.193140 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "865fbd674db34f668fb5a2f5836a9b43" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:34.199379 3007 raft_consensus.cc:493] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.199604 3007 raft_consensus.cc:515] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.200438 3007 leader_election.cc:290] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:34.204759 3227 tablet.cc:1722] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251026 02:56:34.208557 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 277554us
I20251026 02:56:34.213452 1915 leader_election.cc:304] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:34.214116 3163 raft_consensus.cc:2804] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:34.214277 3163 raft_consensus.cc:697] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:34.214586 3163 consensus_queue.cc:237] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:34.217628 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9cd10092aac40068a1c7f85e514fd37" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:34.218464 2259 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.018s sys 0.020s
W20251026 02:56:34.218541 2053 leader_election.cc:343] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:34.218611 2259 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.503s user 0.018s sys 0.020s
I20251026 02:56:34.219583 3007 raft_consensus.cc:493] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.219830 3007 raft_consensus.cc:515] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.220813 3007 leader_election.cc:290] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:34.224104 2381 tablet_bootstrap.cc:492] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:34.224404 2381 ts_tablet_manager.cc:1403] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.319s user 0.005s sys 0.023s
W20251026 02:56:34.228771 2053 leader_election.cc:343] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:34.228920 2053 leader_election.cc:304] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:34.229466 3007 raft_consensus.cc:2749] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:34.238281 3250 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251026 02:56:34.246876 2053 leader_election.cc:343] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:34.247026 2053 leader_election.cc:304] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:34.247560 3007 raft_consensus.cc:2749] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:34.252629 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fba4c45dc0448fc95469df5be374809. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.254222 2381 raft_consensus.cc:359] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.254483 2381 raft_consensus.cc:385] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:34.255062 2375 tablet_bootstrap.cc:492] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:34.255240 2375 ts_tablet_manager.cc:1403] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.095s user 0.002s sys 0.002s
I20251026 02:56:34.256210 2375 raft_consensus.cc:359] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.262152 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 9ec035b1caac450d9ab55e74a061ae38 (table default.Table_9cfeb6e20450444da7688fe216f7a5c0 [id=e905acf634364d5896dc4cfbbbb27f77]) successfully deleted
W20251026 02:56:34.262900 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 already in progress: creating tablet
W20251026 02:56:34.263754 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet dbf2a1ce78154415b23ab6d0efbc1f52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbf2a1ce78154415b23ab6d0efbc1f52 already in progress: creating tablet
I20251026 02:56:34.266209 3163 raft_consensus.cc:493] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.266486 3163 raft_consensus.cc:515] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.267335 3246 raft_consensus.cc:493] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.267469 3246 raft_consensus.cc:515] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.267558 3163 leader_election.cc:290] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:34.268180 3246 leader_election.cc:290] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:34.268316 2375 raft_consensus.cc:385] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:34.268430 2375 raft_consensus.cc:740] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:34.268698 2375 consensus_queue.cc:260] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.268842 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "548a7f9628ac4623b2c2f723f83c1e45" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:34.269881 2375 ts_tablet_manager.cc:1434] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20251026 02:56:34.269853 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbf2a1ce78154415b23ab6d0efbc1f52" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:34.270622 2375 tablet_bootstrap.cc:492] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:34.273164 2375 tablet_bootstrap.cc:654] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:34.276341 2053 leader_election.cc:343] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:34.289314 2375 tablet_bootstrap.cc:492] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:34.289608 2375 ts_tablet_manager.cc:1403] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20251026 02:56:34.290776 2375 raft_consensus.cc:359] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.291119 2375 raft_consensus.cc:385] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:34.291234 2375 raft_consensus.cc:740] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:34.291568 2375 consensus_queue.cc:260] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.292115 2378 tablet_bootstrap.cc:492] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:34.292315 2378 ts_tablet_manager.cc:1403] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.238s user 0.006s sys 0.006s
I20251026 02:56:34.293795 2378 raft_consensus.cc:359] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.294157 2378 raft_consensus.cc:385] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:34.294262 2378 raft_consensus.cc:740] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:34.294667 2378 consensus_queue.cc:260] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.254722 2381 raft_consensus.cc:740] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:34.295619 2375 ts_tablet_manager.cc:1434] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251026 02:56:34.295969 2381 consensus_queue.cc:260] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.296778 2381 ts_tablet_manager.cc:1434] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.044s user 0.001s sys 0.001s
I20251026 02:56:34.297149 2381 tablet_bootstrap.cc:492] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:34.301695 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fe0e06a86e241bc91a1db4219fe25bc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:34.304308 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e62064fbdffb4954a3a1227efd6327ee" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.304677 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "865fbd674db34f668fb5a2f5836a9b43" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.304942 2375 tablet_bootstrap.cc:492] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:34.305030 2259 raft_consensus.cc:2468] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:34.304388 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1370b9ad917344c093e5da3d0e9b1cf0" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.305955 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8688a0bf946841deb5d88c931538b129" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.306042 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "548a7f9628ac4623b2c2f723f83c1e45" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.305955 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44af74033ccf4d048475f12d164938f5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.306950 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9cd10092aac40068a1c7f85e514fd37" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.307159 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dba102838945453cbf44e1cfe505a52c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.307583 2375 tablet_bootstrap.cc:654] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:34.308362 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbf2a1ce78154415b23ab6d0efbc1f52" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:34.314422 2194 leader_election.cc:343] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:34.314963 3245 raft_consensus.cc:493] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.315109 3245 raft_consensus.cc:515] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:34.315614 2194 consensus_peers.cc:597] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:34.316956 1700 master_runner.cc:186] Error getting master registration for 127.1.150.60:33729: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111),
I20251026 02:56:34.317150 1700 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 02:56:34.315641 3245 leader_election.cc:290] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:34.315788 3150 raft_consensus.cc:493] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.317610 3150 raft_consensus.cc:515] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.318171 3150 leader_election.cc:290] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:34.321936 2372 ts_tablet_manager.cc:1950] T a111848706b743c9beb2fbd21f59d7f3 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:34.322853 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 496) took 3137 ms. Trace:
I20251026 02:56:34.322942 2372 rpcz_store.cc:276] 1026 02:56:31.185524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.185793 (+ 269us) service_pool.cc:225] Handling call
1026 02:56:33.827844 (+2642051us) ts_tablet_manager.cc:1151] Deleting tablet a111848706b743c9beb2fbd21f59d7f3
1026 02:56:33.827854 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:34.037287 (+209433us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:34.142048 (+104761us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:34.322831 (+180783us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2641773}
I20251026 02:56:34.323299 2372 tablet_replica.cc:333] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:34.323483 2372 raft_consensus.cc:2243] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:34.323716 2372 raft_consensus.cc:2272] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:34.324667 3144 raft_consensus.cc:515] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.325258 3144 leader_election.cc:290] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:34.326460 3190 tablet.cc:1722] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20251026 02:56:34.326941 3235 log.cc:927] Time spent T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
I20251026 02:56:34.328859 2381 tablet_bootstrap.cc:654] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:34.328814 2195 leader_election.cc:304] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:34.329903 3150 raft_consensus.cc:2804] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:34.330008 3150 raft_consensus.cc:697] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:34.330225 3150 consensus_queue.cc:237] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.330657 3206 tablet.cc:1722] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:34.331597 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 82e4bc6195484074b0da610a8d1b65bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:34.334697 2195 consensus_peers.cc:597] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22e4a433801a4cb58062d3e76c760b69. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:34.335862 3144 consensus_queue.cc:1048] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:34.336560 3234 log.cc:927] Time spent T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20251026 02:56:34.336679 1662 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 046a34f9408b406ba08471421f422064 (table default.Table_760b799491d7426a86aa1dc3c76fab31 [id=bc8a2d95ed134fae9f3e9deeac4f130a]). Expected version 3 got 0
I20251026 02:56:34.337164 2372 ts_tablet_manager.cc:1916] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:34.338793 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1abe312e689149299920c58ec6f1a35d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:34.339484 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1abe312e689149299920c58ec6f1a35d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.340152 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_632f764fc25e4762ad2de1c2a34cf713" } modify_external_catalogs: true
I20251026 02:56:34.340364 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_632f764fc25e4762ad2de1c2a34cf713" } modify_external_catalogs: true
I20251026 02:56:34.341672 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7a84416153d6424eb94a56932605f7ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:34.342182 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a84416153d6424eb94a56932605f7ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.342895 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_908a8748249c46ad964acc6abf54cd58" } modify_external_catalogs: true
I20251026 02:56:34.343052 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_908a8748249c46ad964acc6abf54cd58" } modify_external_catalogs: true
I20251026 02:56:34.344312 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_51306871ca71474db5412b80acf6f5e7" } new_table_name: "default.Table_6ec311efd79c42aeaf08d5bfa17ab79f" modify_external_catalogs: true
I20251026 02:56:34.345758 1663 catalog_manager.cc:5649] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:34.346182 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 352a284902c94f088e2e6ae006162db0 (table default.Table_6cd4ee82e9ec406f971669b7f0722677 [id=414375003cd44b5c8c9f0249b339c28a]). Expected version 1 got 0
I20251026 02:56:34.346844 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fe0e06a86e241bc91a1db4219fe25bc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:34.347014 2114 raft_consensus.cc:2468] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:34.347470 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7ece114e4cfd4133871448f9ea3ce9e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:34.347832 2258 raft_consensus.cc:2468] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
W20251026 02:56:34.347980 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ece114e4cfd4133871448f9ea3ce9e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.358036 2375 tablet_bootstrap.cc:492] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:34.358227 2375 ts_tablet_manager.cc:1403] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.002s
I20251026 02:56:34.359277 2375 raft_consensus.cc:359] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:34.359498 2375 raft_consensus.cc:385] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:34.359556 2375 raft_consensus.cc:740] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:34.359793 2375 consensus_queue.cc:260] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:34.360230 3007 raft_consensus.cc:493] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.360375 3007 raft_consensus.cc:515] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.360899 3007 leader_election.cc:290] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:34.360877 2375 ts_tablet_manager.cc:1434] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:34.363418 2375 tablet_bootstrap.cc:492] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:34.365074 2375 tablet_bootstrap.cc:654] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:34.370632 3259 log.cc:927] Time spent T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
W20251026 02:56:34.372308 1944 consensus_meta.cc:287] T 82e4bc6195484074b0da610a8d1b65bc P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.631s user 0.004s sys 0.000s
I20251026 02:56:34.372155 2378 ts_tablet_manager.cc:1434] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
W20251026 02:56:34.372411 1943 consensus_meta.cc:287] T f15000847df44f1ebebad3ca2c2b8046 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.651s user 0.005s sys 0.000s
W20251026 02:56:34.372524 1952 consensus_meta.cc:287] T 58d60bf2a9e546e6a9474a5af6d5273a P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.626s user 0.005s sys 0.001s
W20251026 02:56:34.372182 1949 consensus_meta.cc:287] T 1ffead7ca2d946c78aa4b8a602865fc8 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.623s user 0.005s sys 0.000s
W20251026 02:56:34.373570 1950 consensus_meta.cc:287] T cedff3287c7b4a07a81462e8c4b44970 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.671s user 0.004s sys 0.000s
I20251026 02:56:34.374853 1974 raft_consensus.cc:3060] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:34.372191 1955 consensus_meta.cc:287] T 657e07d0b17c44d4b78b97210f17ce40 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.611s user 0.004s sys 0.000s
I20251026 02:56:34.376195 2375 tablet_bootstrap.cc:492] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:34.376459 2375 ts_tablet_manager.cc:1403] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20251026 02:56:34.377665 2375 raft_consensus.cc:359] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.377983 2375 raft_consensus.cc:385] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:34.378062 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 6699b50d87964f6a89c365171253ad25
I20251026 02:56:34.378134 2375 raft_consensus.cc:740] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:34.378419 2375 consensus_queue.cc:260] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.379695 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 5824831b8e184daf8b40543686c06467
I20251026 02:56:34.373762 3163 raft_consensus.cc:493] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.380545 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 848e414f6e134761818039f7e98500ac
I20251026 02:56:34.381461 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 470) took 3152 ms. Trace:
I20251026 02:56:34.381567 2376 rpcz_store.cc:276] 1026 02:56:31.171809 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.171878 (+ 69us) service_pool.cc:225] Handling call
1026 02:56:33.423020 (+2251142us) ts_tablet_manager.cc:1151] Deleting tablet c570e5f9a93c45bda32ec92c68794714
1026 02:56:33.423031 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:33.475816 (+ 52785us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:33.738149 (+262333us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:34.324453 (+586304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2250880}
I20251026 02:56:34.381911 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 5ddc7972f29a429ea51f6c6c1a074baf
I20251026 02:56:34.381953 2376 tablet_replica.cc:333] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:34.382133 2376 raft_consensus.cc:2243] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:34.382314 2376 raft_consensus.cc:2272] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:34.382498 1946 consensus_meta.cc:287] T 4ed205c1800243ed9830d720ac85de2f P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.634s user 0.002s sys 0.003s
I20251026 02:56:34.382869 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 358e805370ef410e9d65ee92a8d04f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:34.383060 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b0406ceb60b48969ca84f6268b43dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:34.383414 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 4b396cec093c4f40844b901550eee1f5 (DEFAULT_TABLE table=default.Table_8929007b52484dcca450eb9ea54fe75d [id=6ce21ddef8314008a1e55705ed6c1358]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.383858 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b396cec093c4f40844b901550eee1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.384579 2375 ts_tablet_manager.cc:1434] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20251026 02:56:34.385015 1957 consensus_meta.cc:287] T b4005a095c594469b8d8105a0679ff45 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.645s user 0.006s sys 0.000s
I20251026 02:56:34.385598 1957 tablet_service.cc:1505] Processing CreateTablet for tablet a956ebfb1eaa4d6fb1575b4165293193 (DEFAULT_TABLE table=default.Table_9b0942af08414e7bb088a14c781f0edd [id=5fb61e88f7074976bac29b40f4466ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.385970 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a956ebfb1eaa4d6fb1575b4165293193. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:34.386639 1954 consensus_meta.cc:287] T 1bed60c80d584cba812f045d94995098 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.646s user 0.006s sys 0.000s
I20251026 02:56:34.387272 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 1fba4c45dc0448fc95469df5be374809 (DEFAULT_TABLE table=default.Table_472acbcc3a244422abe7a7477c65c87a [id=fb698813057b43cd940cadac51367204]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.387646 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fba4c45dc0448fc95469df5be374809. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.387616 3006 raft_consensus.cc:493] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.387737 3006 raft_consensus.cc:515] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.388365 3006 leader_election.cc:290] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:34.391002 3006 raft_consensus.cc:493] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.391172 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet a111848706b743c9beb2fbd21f59d7f3 (table default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]) successfully deleted
I20251026 02:56:34.391148 3006 raft_consensus.cc:515] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.391687 3006 leader_election.cc:290] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:34.392534 3006 raft_consensus.cc:493] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.392647 3006 raft_consensus.cc:515] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.393114 3006 leader_election.cc:290] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:34.393551 1952 tablet_service.cc:1505] Processing CreateTablet for tablet a29ccd3f7b204cccb8c82d1589299350 (DEFAULT_TABLE table=default.Table_5b95c2f1040441baa94ad346e22b4c72 [id=c82e2aa4eced4a8da31014e4b768ab2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.393546 2375 tablet_bootstrap.cc:492] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:34.393967 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29ccd3f7b204cccb8c82d1589299350. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:34.394946 3256 log.cc:927] Time spent T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20251026 02:56:34.396288 1949 tablet_service.cc:1505] Processing CreateTablet for tablet 59f5078f03224ef18ef4b6864be7122b (DEFAULT_TABLE table=default.Table_f28a7b90302c4e3993516e647d2bc67c [id=49086aec5a2b49bbb54e04f181f6c7fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.396672 1949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59f5078f03224ef18ef4b6864be7122b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.396636 3163 raft_consensus.cc:515] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:34.397331 3163 leader_election.cc:290] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:34.397492 1953 consensus_meta.cc:287] T 9b0406ceb60b48969ca84f6268b43dbb P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.680s user 0.001s sys 0.003s
I20251026 02:56:34.397657 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 5d7261ee6f244e77891e37a4c1a984ed (DEFAULT_TABLE table=default.Table_d1d2850518cc4c51a5cd3296d2d19680 [id=e6b5e81aa90d44a79a7994bb4d4877ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.398065 1953 tablet_service.cc:1552] Processing DeleteTablet for tablet 82e4bc6195484074b0da610a8d1b65bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:34.398285 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d7261ee6f244e77891e37a4c1a984ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.398201 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "358e805370ef410e9d65ee92a8d04f34" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:34.398509 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 2ea41342e01f4b57ada0d40379b133ea (DEFAULT_TABLE table=default.Table_d52e76dbc27d4816bb3346700e4b4e8b [id=0259347470054a328444a36d2dff52b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.398859 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea41342e01f4b57ada0d40379b133ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.399339 1944 tablet_service.cc:1505] Processing CreateTablet for tablet d9391e7654c442a5bb312f0daaf1d99b (DEFAULT_TABLE table=default.Table_2e3bf8087ce7480584b9b66c8270c451 [id=0f4a4be6bea7411aa2e13984864e827c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.399693 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9391e7654c442a5bb312f0daaf1d99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.400749 2376 ts_tablet_manager.cc:1916] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:34.401995 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 7062c5ff91e14369ab69a2d199585c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:34.402218 2378 tablet_bootstrap.cc:492] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:34.402438 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 6cbea840b88c4ef6b140a9a769fd69df (DEFAULT_TABLE table=default.Table_4996540a4e3749c6aa9bdecf10b2fae3 [id=ffe25b8112e447fa85edc84784f9c95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.402799 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cbea840b88c4ef6b140a9a769fd69df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.406781 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 2ea41342e01f4b57ada0d40379b133ea (DEFAULT_TABLE table=default.Table_d52e76dbc27d4816bb3346700e4b4e8b [id=0259347470054a328444a36d2dff52b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.407985 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea41342e01f4b57ada0d40379b133ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.410282 2375 tablet_bootstrap.cc:654] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:34.412979 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet d212170f056a44dfb15cfcb7f9fbae2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:34.413496 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 086b40036f0c4c97bb95647f91ce7939 (DEFAULT_TABLE table=default.Table_c4c451a485f24525b3dbb3f3626a01e7 [id=5225751c079f4b7cbd21bc817c91851e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.417430 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086b40036f0c4c97bb95647f91ce7939. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.418220 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 6cbea840b88c4ef6b140a9a769fd69df (DEFAULT_TABLE table=default.Table_4996540a4e3749c6aa9bdecf10b2fae3 [id=ffe25b8112e447fa85edc84784f9c95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.418684 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cbea840b88c4ef6b140a9a769fd69df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.419757 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet c09153fbfe18434393c188058a3461d5
I20251026 02:56:34.394840 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f3ae3a377af46449c3faca339f86053" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:34.422295 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ec5b59f3b554525a0e1a7d57ec2962f consensus state has no leader
W20251026 02:56:34.423959 2053 leader_election.cc:343] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:34.424109 2053 leader_election.cc:304] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:34.424571 2053 leader_election.cc:343] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:34.424669 2053 leader_election.cc:304] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:34.425212 3007 raft_consensus.cc:2749] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:34.425367 3007 raft_consensus.cc:2749] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:34.425490 3007 consensus_queue.cc:1048] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:34.428166 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efd751ee4b1d433198dcd00c081f648d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:34.428393 2116 raft_consensus.cc:2468] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:34.428665 2115 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7da859119bd42ea9fa1290b3d4bf88c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:34.428839 2115 raft_consensus.cc:2468] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:34.429088 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2524ee0c964345459c3e62f1e39b8f9e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:34.429230 2120 raft_consensus.cc:3060] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:34.429530 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet e9cd10092aac40068a1c7f85e514fd37 (table default.Table_fe09be2527a840b09e3e64c27965845d [id=de5272bcc29644208ddd95bdec21254e]). Expected version 1 got 0
I20251026 02:56:34.430052 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet c4b0933f71cb4ac7a58986d16b613e64 (table default.Table_056ac0729a274fb19574d75c2f37511b [id=cba6d4517d244d7580e525729e871745]). Expected version 2 got 0
I20251026 02:56:34.431578 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3afd36d31ad34be682d7a49ab29da62b" } new_table_name: "default.Table_8dadea9fa7254668b84991c8d7cc4f52" modify_external_catalogs: true
I20251026 02:56:34.433229 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_901960ec4b4749f7b73a558f65408857"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:34.433832 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_901960ec4b4749f7b73a558f65408857 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.435496 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_852e528c9c8c460d82d9e34569c8860d" } new_table_name: "default.Table_afa20b14ea2b4ebab253b18ecca40a69" modify_external_catalogs: true
I20251026 02:56:34.435916 2258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "358e805370ef410e9d65ee92a8d04f34" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.436842 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_dcdbf613e1704183b39ba6294a0b202d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:34.437443 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcdbf613e1704183b39ba6294a0b202d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.438694 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f5e1fc5d0a604a82be231f8b70013315" } new_table_name: "default.Table_5e627b2d763041a788ce88beb39f25d6" modify_external_catalogs: true
I20251026 02:56:34.439143 2120 raft_consensus.cc:2468] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:34.445947 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 086b40036f0c4c97bb95647f91ce7939 (DEFAULT_TABLE table=default.Table_c4c451a485f24525b3dbb3f3626a01e7 [id=5225751c079f4b7cbd21bc817c91851e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.447103 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086b40036f0c4c97bb95647f91ce7939. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:34.449002 3028 consensus_meta.cc:287] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539: Time spent flushing consensus metadata: real 0.538s user 0.001s sys 0.002s
I20251026 02:56:34.454541 2258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f3ae3a377af46449c3faca339f86053" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.455013 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741792cf96164695b03de3fb86b12602" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.458030 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 106524801a204a81b9f194321768f433 (DEFAULT_TABLE table=default.Table_11f1980aeb6a4f38a5c9eebc431ddd86 [id=a3113643104d49949c503729cb7d87a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.458478 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 106524801a204a81b9f194321768f433. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.449131 3028 raft_consensus.cc:515] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:34.458971 1661 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_73789469dbaa4885b8566a3aa0877893 [id=9e897d33e6374ad4bc3c8acaae512645] alter complete (version 2)
I20251026 02:56:34.466527 2381 tablet_bootstrap.cc:492] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:34.466674 2381 ts_tablet_manager.cc:1403] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.170s user 0.002s sys 0.002s
I20251026 02:56:34.467367 2381 raft_consensus.cc:359] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.467527 2381 raft_consensus.cc:385] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:34.467574 2381 raft_consensus.cc:740] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:34.467734 2381 consensus_queue.cc:260] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.468194 2381 ts_tablet_manager.cc:1434] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251026 02:56:34.468864 2381 tablet_bootstrap.cc:492] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:34.470364 2381 tablet_bootstrap.cc:654] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:34.472476 1915 leader_election.cc:304] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
W20251026 02:56:34.478060 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4f4d7f87916425999574fd5b9f2fdf7 consensus state has no leader
W20251026 02:56:34.478658 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c966cae6886048a2b6ad3b8bd7007bcb consensus state has no leader
W20251026 02:56:34.478669 1960 consensus_meta.cc:287] T e044af9739364ecdbbaddc115c09bfab P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.681s user 0.002s sys 0.001s
W20251026 02:56:34.479140 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cedff3287c7b4a07a81462e8c4b44970 consensus state has no leader
W20251026 02:56:34.480391 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 47640.3us
I20251026 02:56:34.483712 3028 leader_election.cc:290] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:34.485242 3163 raft_consensus.cc:2749] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251026 02:56:34.487089 1948 consensus_meta.cc:287] T 144b957c69e64085a6f78050dea426b4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.617s user 0.003s sys 0.001s
W20251026 02:56:34.490722 1649 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 4ec5b59f3b554525a0e1a7d57ec2962f (table default.Table_6ec311efd79c42aeaf08d5bfa17ab79f [id=10bab83b438b4584ab88a7f1241809c0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4ec5b59f3b554525a0e1a7d57ec2962f consensus state has no leader
I20251026 02:56:34.492798 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae95654d129441e08fcb3f3b355ffc22" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.492885 2195 leader_election.cc:304] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:34.493106 3252 consensus_queue.cc:1048] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:34.493384 3144 raft_consensus.cc:2804] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:34.493451 3144 raft_consensus.cc:493] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:34.493532 3144 raft_consensus.cc:3060] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:34.494135 3227 tablet.cc:1722] T 740f7c7f186f45ceaa4c40e76419e3a1 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251026 02:56:34.494509 2378 tablet_bootstrap.cc:654] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:34.498020 3065 consensus_queue.cc:1048] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:34.502420 1950 tablet_service.cc:1505] Processing CreateTablet for tablet a7a5020acff04c1d8b5749cdfd7dddfd (DEFAULT_TABLE table=default.Table_6798c68a24044bce9d44d8f21417ebf0 [id=f486f04ebeb34588921c061f9322a2a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.507009 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "046a34f9408b406ba08471421f422064" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:34.507107 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 345ddc32a45544e0b4cb3bab5f3ed1bf (DEFAULT_TABLE table=default.Table_42d878cfb9464b9a8403399260da8d43 [id=4c81ac27972f4d5a82af29ce7d647238]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.507510 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 345ddc32a45544e0b4cb3bab5f3ed1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.509019 1915 leader_election.cc:304] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:34.509488 1915 leader_election.cc:304] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:34.509907 1915 leader_election.cc:304] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:34.510502 3166 raft_consensus.cc:2804] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:34.510637 3166 raft_consensus.cc:493] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:34.510746 3166 raft_consensus.cc:3060] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:34.513346 3166 raft_consensus.cc:515] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:34.514575 3166 leader_election.cc:290] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:34.514775 3166 raft_consensus.cc:2749] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:34.515010 3166 raft_consensus.cc:2749] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:34.515404 1951 consensus_meta.cc:287] T c09153fbfe18434393c188058a3461d5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.628s user 0.003s sys 0.000s
I20251026 02:56:34.515956 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 2bba8975f7a148ecae245bd7381ba436 (DEFAULT_TABLE table=default.Table_9c462c3663254348a3d6f661a26cd0bc [id=d6b7723e2b20456482c767e861471a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.522399 2377 ts_tablet_manager.cc:1929] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:34.522538 2377 log.cc:1199] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/9a08e8b7eb1949408f2151f2d7257e9d
I20251026 02:56:34.524168 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 345ddc32a45544e0b4cb3bab5f3ed1bf (DEFAULT_TABLE table=default.Table_42d878cfb9464b9a8403399260da8d43 [id=4c81ac27972f4d5a82af29ce7d647238]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.529889 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 106524801a204a81b9f194321768f433 (DEFAULT_TABLE table=default.Table_11f1980aeb6a4f38a5c9eebc431ddd86 [id=a3113643104d49949c503729cb7d87a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.530467 2377 ts_tablet_manager.cc:1950] T 9a08e8b7eb1949408f2151f2d7257e9d P 696476265495492790739ec126a08539: Deleting consensus metadata
W20251026 02:56:34.532020 1959 consensus_meta.cc:287] T 31edacb952514c8aa25c79f509869e8f P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.705s user 0.004s sys 0.000s
I20251026 02:56:34.532672 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 68ad3d4a2d484badb4b35884517b72f1 (DEFAULT_TABLE table=default.Table_890ffb18fe5a49f1938a0fca64189907 [id=5e42165433a248a9bff99b74846d72e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.540271 3144 raft_consensus.cc:515] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:34.540982 1956 consensus_meta.cc:287] T 52a1312bf5d044b1b21454f6cefe7980 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.796s user 0.002s sys 0.000s
I20251026 02:56:34.541186 3144 leader_election.cc:290] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:34.544935 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (table default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5]). Expected version 1 got 0
I20251026 02:56:34.545250 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 5226d4516ce24ecda51dab35c86da132 (table default.Table_5221808b553f44bc82646e59a4f6881e [id=c8435cef755d4de69c33f599df1c0f75]). Expected version 3 got 2
W20251026 02:56:34.551338 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 52766us
I20251026 02:56:34.551738 2195 leader_election.cc:304] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:34.552013 2962 raft_consensus.cc:2804] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:34.552093 2962 raft_consensus.cc:493] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:34.552170 2962 raft_consensus.cc:3060] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:34.552873 1962 consensus_meta.cc:287] T 46547883cfe041f3954aa4128fba1d8c P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.691s user 0.004s sys 0.000s
I20251026 02:56:34.554839 2962 raft_consensus.cc:515] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.555437 2962 leader_election.cc:290] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:34.555572 2962 consensus_queue.cc:1048] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:34.555915 2962 consensus_queue.cc:1048] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:34.556906 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 452) took 5293 ms. Trace:
I20251026 02:56:34.556985 2377 rpcz_store.cc:276] 1026 02:56:29.263257 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.263304 (+ 47us) service_pool.cc:225] Handling call
1026 02:56:34.130258 (+4866954us) ts_tablet_manager.cc:1151] Deleting tablet 9a08e8b7eb1949408f2151f2d7257e9d
1026 02:56:34.130268 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:34.439955 (+309687us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:34.522267 (+ 82312us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:34.556883 (+ 34616us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23662,"tablet-delete.queue_time_us":4866760}
I20251026 02:56:34.557432 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 448) took 5311 ms. Trace:
I20251026 02:56:34.557504 2377 rpcz_store.cc:276] 1026 02:56:29.245677 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.245738 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:34.557299 (+5311561us) ts_tablet_manager.cc:1151] Deleting tablet b1ca5b8379c94deab1130bfe8bdc87b0
1026 02:56:34.557321 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:34.557418 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5287765}
I20251026 02:56:34.557750 2377 tablet_replica.cc:333] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539: stopping tablet replica
W20251026 02:56:34.557760 1945 consensus_meta.cc:287] T 5aff38ba32864ad18b5d808e5235efe3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.706s user 0.004s sys 0.001s
I20251026 02:56:34.557951 2377 raft_consensus.cc:2243] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:34.558135 2377 pending_rounds.cc:70] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:34.558182 2377 pending_rounds.cc:77] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888523555131392 op_type: NO_OP noop_request { }
I20251026 02:56:34.558215 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 5824831b8e184daf8b40543686c06467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:34.558264 2377 raft_consensus.cc:2889] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:34.558348 2377 raft_consensus.cc:2272] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:34.558430 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 848e414f6e134761818039f7e98500ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:34.558797 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 (DEFAULT_TABLE table=default.Table_61f1d10f16b1467aaab14fe4fbaef1bc [id=216da41d45b24046b44678c1ff34c17e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.558892 2377 ts_tablet_manager.cc:1916] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:34.552172 2195 leader_election.cc:304] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:34.561384 3065 raft_consensus.cc:2804] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:34.561458 3065 raft_consensus.cc:493] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:34.561528 3065 raft_consensus.cc:3060] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:34.562008 1958 tablet_service.cc:1505] Processing CreateTablet for tablet d01fa8b270e1476cb86d81a196ba94f8 (DEFAULT_TABLE table=default.Table_d9351d4a858248d8813e9df05f900010 [id=06e6f90503e344cb993a0fb798f690bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.564203 3065 raft_consensus.cc:515] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.567114 3065 leader_election.cc:290] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:34.568832 2195 leader_election.cc:304] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
W20251026 02:56:34.569236 1947 consensus_meta.cc:287] T c2ca394b096f4fd88f8503488b08d8d5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.673s user 0.003s sys 0.000s
I20251026 02:56:34.570472 2377 ts_tablet_manager.cc:1929] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:34.570622 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 19016e8e11cc42c69ce44a819db9e298 (DEFAULT_TABLE table=default.Table_fda5e7b00a8a4e9ba95382d569d8c421 [id=b10cfa3fcea04a5cb33be14a094a1482]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.571717 2084 tablet_service.cc:1505] Processing CreateTablet for tablet d01fa8b270e1476cb86d81a196ba94f8 (DEFAULT_TABLE table=default.Table_d9351d4a858248d8813e9df05f900010 [id=06e6f90503e344cb993a0fb798f690bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.572118 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d01fa8b270e1476cb86d81a196ba94f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.572700 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 6699b50d87964f6a89c365171253ad25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:34.573000 1916 leader_election.cc:343] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: e62064fbdffb4954a3a1227efd6327ee
W20251026 02:56:34.573153 1916 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 50327.2us
I20251026 02:56:34.580473 2081 tablet_service.cc:1505] Processing CreateTablet for tablet a7a5020acff04c1d8b5749cdfd7dddfd (DEFAULT_TABLE table=default.Table_6798c68a24044bce9d44d8f21417ebf0 [id=f486f04ebeb34588921c061f9322a2a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.580977 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a5020acff04c1d8b5749cdfd7dddfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.582878 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 7e1a45412db1449a99252112d8110a17 (DEFAULT_TABLE table=default.Table_fa3b58cf82f242cebd20cc1ea9b2133d [id=4a727b205b89448fb97b6bde0c3dd210]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.492605 2372 ts_tablet_manager.cc:1929] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:34.586998 2372 log.cc:1199] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/276bd3b97c97426daf802e1f57feb265
I20251026 02:56:34.587484 2372 ts_tablet_manager.cc:1950] T 276bd3b97c97426daf802e1f57feb265 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:34.588380 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 499) took 3288 ms. Trace:
I20251026 02:56:34.588534 2372 rpcz_store.cc:276] 1026 02:56:31.299606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.299675 (+ 69us) service_pool.cc:225] Handling call
1026 02:56:34.323243 (+3023568us) ts_tablet_manager.cc:1151] Deleting tablet 276bd3b97c97426daf802e1f57feb265
1026 02:56:34.323252 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:34.363610 (+ 40358us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:34.462521 (+ 98911us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:34.588359 (+125838us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3023303}
I20251026 02:56:34.589066 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 501) took 3223 ms. Trace:
I20251026 02:56:34.589190 2372 rpcz_store.cc:276] 1026 02:56:31.365900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.366094 (+ 194us) service_pool.cc:225] Handling call
1026 02:56:34.588926 (+3222832us) ts_tablet_manager.cc:1151] Deleting tablet 4b5e61afe90e4bc4a2897de21e0b3919
1026 02:56:34.588937 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:34.589050 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3222551}
I20251026 02:56:34.583411 1947 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ddc7972f29a429ea51f6c6c1a074baf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:34.586845 1961 consensus_meta.cc:287] T 3efafaac6c634d088396d56c8debd990 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.727s user 0.002s sys 0.000s
I20251026 02:56:34.586918 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 8977f824ff374d05b4ed9d88c67329f9 (DEFAULT_TABLE table=default.Table_7d238b86df1246ffa4b5d4bb9af0f387 [id=ce9076501c55430a9e47e5ff3b07c5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.587116 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 4ec5b59f3b554525a0e1a7d57ec2962f (DEFAULT_TABLE table=default.Table_51306871ca71474db5412b80acf6f5e7 [id=10bab83b438b4584ab88a7f1241809c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.587203 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 31cae8e4448549fd827f2bf836d9634d (DEFAULT_TABLE table=default.Table_c4160ae9bbe94aa8a45dbf23888f7aaf [id=bc99d7ca957d43589c58153b199f45ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.587342 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a5020acff04c1d8b5749cdfd7dddfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.612628 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec5b59f3b554525a0e1a7d57ec2962f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.589783 3250 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:34.613722 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bba8975f7a148ecae245bd7381ba436. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.614223 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.594645 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 7f5cd37c87f34a40bf05829af6de0473 (DEFAULT_TABLE table=default.Table_d450815506d645aab06dee257de7ea3e [id=f04a81a5e5ec403d8533855c82814d07]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:34.596434 3240 log.cc:927] Time spent T 47bfff40c181432896d6898ea50cd025 P 7fef01f4f2b843538ea245a09dd087a8: Append to log took a long time: real 0.312s user 0.000s sys 0.000s
I20251026 02:56:34.597353 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 0efa5bb6b73e493fa53f549522aadb33 (DEFAULT_TABLE table=default.Table_5322450167b9484c9e1448d711dc9702 [id=28ea5b74b9f848ad9e1531de21ab2cd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.597543 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 1219e8bdfde9444f9bfdf1d626389e88 (DEFAULT_TABLE table=default.Table_aca29902287d4d8fa34e85821bfa93db [id=a7b8f1c1554d44e1b2aceff82f0771f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.598809 1949 tablet_service.cc:1505] Processing CreateTablet for tablet 7c82a43e97cd4f0a9cb24c598238d849 (DEFAULT_TABLE table=default.Table_d48a43604ff140f0b7103cc9fb517a5c [id=1da8a6913ca44ef7b03a7d87e41e1e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.599071 3163 raft_consensus.cc:493] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:34.624106 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (table default.Table_1fa437ef7f89450bb6f672a1fb491baa [id=44c37945ff9f48e8b2c07648394ac177]). Expected version 1 got 0
I20251026 02:56:34.632562 1947 tablet_service.cc:1505] Processing CreateTablet for tablet 8ae2fa984c454a199e12eac3c0ed6684 (DEFAULT_TABLE table=default.Table_c76a8a4746f9463eb166591b7c7b5713 [id=8259cee436bd46ed9cc0c0a3598044ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.615111 3163 raft_consensus.cc:515] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:34.604658 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 19016e8e11cc42c69ce44a819db9e298 (DEFAULT_TABLE table=default.Table_fda5e7b00a8a4e9ba95382d569d8c421 [id=b10cfa3fcea04a5cb33be14a094a1482]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.605327 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 8977f824ff374d05b4ed9d88c67329f9 (DEFAULT_TABLE table=default.Table_7d238b86df1246ffa4b5d4bb9af0f387 [id=ce9076501c55430a9e47e5ff3b07c5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.634251 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8977f824ff374d05b4ed9d88c67329f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.634717 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 106524801a204a81b9f194321768f433. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.635116 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31cae8e4448549fd827f2bf836d9634d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.635962 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d01fa8b270e1476cb86d81a196ba94f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.605401 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 2bba8975f7a148ecae245bd7381ba436 (DEFAULT_TABLE table=default.Table_9c462c3663254348a3d6f661a26cd0bc [id=d6b7723e2b20456482c767e861471a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.636508 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bba8975f7a148ecae245bd7381ba436. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.636672 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 345ddc32a45544e0b4cb3bab5f3ed1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.637598 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f5cd37c87f34a40bf05829af6de0473. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.638312 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68ad3d4a2d484badb4b35884517b72f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.638854 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8977f824ff374d05b4ed9d88c67329f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.639568 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae2fa984c454a199e12eac3c0ed6684. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.640291 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e1a45412db1449a99252112d8110a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.640933 1961 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58894 (request call id 538) took 1077 ms. Trace:
I20251026 02:56:34.641016 1961 rpcz_store.cc:276] 1026 02:56:33.563853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.564747 (+ 894us) service_pool.cc:225] Handling call
1026 02:56:33.565284 (+ 537us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:33.565319 (+ 35us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:33.793316 (+227997us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:34.611340 (+818024us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:34.640915 (+ 29575us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:34.641513 1961 tablet_service.cc:1505] Processing CreateTablet for tablet bc34ad9deb6d4fcb87d0c0fdfee17bbf (DEFAULT_TABLE table=default.Table_6a273a7631fc410e8ba77d604a7e1c66 [id=1af4863cbdeb4f88a616a450022541b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.605866 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 68ad3d4a2d484badb4b35884517b72f1 (DEFAULT_TABLE table=default.Table_890ffb18fe5a49f1938a0fca64189907 [id=5e42165433a248a9bff99b74846d72e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.641856 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc34ad9deb6d4fcb87d0c0fdfee17bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.642097 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68ad3d4a2d484badb4b35884517b72f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.642409 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0efa5bb6b73e493fa53f549522aadb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.642985 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1219e8bdfde9444f9bfdf1d626389e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.643568 1949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c82a43e97cd4f0a9cb24c598238d849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.605875 2089 tablet_service.cc:1505] Processing CreateTablet for tablet 4ec5b59f3b554525a0e1a7d57ec2962f (DEFAULT_TABLE table=default.Table_51306871ca71474db5412b80acf6f5e7 [id=10bab83b438b4584ab88a7f1241809c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.644788 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 1360fe4278f24f31b4ea3eb28d5abf5d (DEFAULT_TABLE table=default.Table_53a2c0226802442a8954f3903443b574 [id=881224ae789e4daab936faafe4765729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.644989 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec5b59f3b554525a0e1a7d57ec2962f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.645159 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1360fe4278f24f31b4ea3eb28d5abf5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.645984 1946 tablet_service.cc:1505] Processing CreateTablet for tablet ce11081bf85f44e8bf1f92ce0f4bf152 (DEFAULT_TABLE table=default.Table_26257eca63ce4baaa75beae2e9db1340 [id=fbc97817a9584474abc0b2ab2052abb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.646270 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce11081bf85f44e8bf1f92ce0f4bf152. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.606251 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 31cae8e4448549fd827f2bf836d9634d (DEFAULT_TABLE table=default.Table_c4160ae9bbe94aa8a45dbf23888f7aaf [id=bc99d7ca957d43589c58153b199f45ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.648473 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31cae8e4448549fd827f2bf836d9634d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.648540 1658 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_76e2e866c5b84956aac1e1bcde61692e [id=1d09f5b9fa164ac783f0be9332eac4c3] alter complete (version 6)
W20251026 02:56:34.650466 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 225456us
W20251026 02:56:34.650707 2053 leader_election.cc:343] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:34.650800 2053 leader_election.cc:304] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:34.606572 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 6699b50d87964f6a89c365171253ad25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:34.606830 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 5824831b8e184daf8b40543686c06467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:34.607057 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 848e414f6e134761818039f7e98500ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:34.607414 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 (DEFAULT_TABLE table=default.Table_61f1d10f16b1467aaab14fe4fbaef1bc [id=216da41d45b24046b44678c1ff34c17e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.652909 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.653294 1976 raft_consensus.cc:3060] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:34.607964 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 1219e8bdfde9444f9bfdf1d626389e88 (DEFAULT_TABLE table=default.Table_aca29902287d4d8fa34e85821bfa93db [id=a7b8f1c1554d44e1b2aceff82f0771f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.608225 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ddc7972f29a429ea51f6c6c1a074baf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:34.608296 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 0efa5bb6b73e493fa53f549522aadb33 (DEFAULT_TABLE table=default.Table_5322450167b9484c9e1448d711dc9702 [id=28ea5b74b9f848ad9e1531de21ab2cd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.656513 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1219e8bdfde9444f9bfdf1d626389e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.656697 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0efa5bb6b73e493fa53f549522aadb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.608584 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 7f5cd37c87f34a40bf05829af6de0473 (DEFAULT_TABLE table=default.Table_d450815506d645aab06dee257de7ea3e [id=f04a81a5e5ec403d8533855c82814d07]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.661196 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f5cd37c87f34a40bf05829af6de0473. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.608956 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 7e1a45412db1449a99252112d8110a17 (DEFAULT_TABLE table=default.Table_fa3b58cf82f242cebd20cc1ea9b2133d [id=4a727b205b89448fb97b6bde0c3dd210]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.664592 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e1a45412db1449a99252112d8110a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.609462 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 7c82a43e97cd4f0a9cb24c598238d849 (DEFAULT_TABLE table=default.Table_d48a43604ff140f0b7103cc9fb517a5c [id=1da8a6913ca44ef7b03a7d87e41e1e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.665930 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c82a43e97cd4f0a9cb24c598238d849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.669833 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efd751ee4b1d433198dcd00c081f648d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:34.670419 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7da859119bd42ea9fa1290b3d4bf88c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:34.670840 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2524ee0c964345459c3e62f1e39b8f9e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:34.672087 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5caeb11322fe48348ebb523967152187" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:34.673010 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fe0e06a86e241bc91a1db4219fe25bc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:34.673381 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efd751ee4b1d433198dcd00c081f648d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:34.673753 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7da859119bd42ea9fa1290b3d4bf88c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:34.689806 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 9a5670a9e10c4a9db64881e7e14e839e (DEFAULT_TABLE table=default.Table_e250def75d2042ee89b9ce3bd7ed34ab [id=12c3c591a85f4e1187bebde0eb667c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.690366 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a5670a9e10c4a9db64881e7e14e839e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.691298 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19016e8e11cc42c69ce44a819db9e298. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.698616 3007 raft_consensus.cc:2749] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:34.699167 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5caeb11322fe48348ebb523967152187" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:34.699357 2120 raft_consensus.cc:3060] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:34.699673 2115 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efd751ee4b1d433198dcd00c081f648d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:34.699844 2115 raft_consensus.cc:3060] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:34.701308 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fe0e06a86e241bc91a1db4219fe25bc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:34.701452 2118 raft_consensus.cc:3060] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:34.702796 2115 raft_consensus.cc:2468] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:34.703600 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7da859119bd42ea9fa1290b3d4bf88c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:34.703722 2116 raft_consensus.cc:3060] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:34.715415 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19016e8e11cc42c69ce44a819db9e298. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.704200 2118 raft_consensus.cc:2468] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:34.723778 3163 leader_election.cc:290] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:34.737569 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 502) took 3363 ms. Trace:
I20251026 02:56:34.737766 2372 rpcz_store.cc:276] 1026 02:56:31.373824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.373890 (+ 66us) service_pool.cc:225] Handling call
1026 02:56:34.737419 (+3363529us) ts_tablet_manager.cc:1151] Deleting tablet 9a345fa5edbd48d2891b8b40b09cb3f8
1026 02:56:34.737431 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:34.737543 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3363250}
I20251026 02:56:34.738246 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 503) took 3361 ms. Trace:
I20251026 02:56:34.738602 2372 rpcz_store.cc:276] 1026 02:56:31.376407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.376476 (+ 69us) service_pool.cc:225] Handling call
1026 02:56:34.738129 (+3361653us) ts_tablet_manager.cc:1151] Deleting tablet a3af2695c41a4439bc4092b7cb1b2edf
1026 02:56:34.738139 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:34.738231 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3361399}
I20251026 02:56:34.590373 3028 raft_consensus.cc:2804] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:34.591778 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 2ea41342e01f4b57ada0d40379b133ea (DEFAULT_TABLE table=default.Table_d52e76dbc27d4816bb3346700e4b4e8b [id=0259347470054a328444a36d2dff52b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.739584 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea41342e01f4b57ada0d40379b133ea. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:34.738590 1916 leader_election.cc:343] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9e5bb0a6a9f947cf9ce119c1fc91fbf5
I20251026 02:56:34.592353 2242 tablet_service.cc:1505] Processing CreateTablet for tablet a7a5020acff04c1d8b5749cdfd7dddfd (DEFAULT_TABLE table=default.Table_6798c68a24044bce9d44d8f21417ebf0 [id=f486f04ebeb34588921c061f9322a2a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.744748 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a5020acff04c1d8b5749cdfd7dddfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.592695 2238 tablet_service.cc:1505] Processing CreateTablet for tablet 2bba8975f7a148ecae245bd7381ba436 (DEFAULT_TABLE table=default.Table_9c462c3663254348a3d6f661a26cd0bc [id=d6b7723e2b20456482c767e861471a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.748621 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bba8975f7a148ecae245bd7381ba436. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.592658 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 8977f824ff374d05b4ed9d88c67329f9 (DEFAULT_TABLE table=default.Table_7d238b86df1246ffa4b5d4bb9af0f387 [id=ce9076501c55430a9e47e5ff3b07c5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.752540 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8977f824ff374d05b4ed9d88c67329f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.592994 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 19016e8e11cc42c69ce44a819db9e298 (DEFAULT_TABLE table=default.Table_fda5e7b00a8a4e9ba95382d569d8c421 [id=b10cfa3fcea04a5cb33be14a094a1482]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.756574 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19016e8e11cc42c69ce44a819db9e298. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.593268 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 345ddc32a45544e0b4cb3bab5f3ed1bf (DEFAULT_TABLE table=default.Table_42d878cfb9464b9a8403399260da8d43 [id=4c81ac27972f4d5a82af29ce7d647238]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.759826 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 345ddc32a45544e0b4cb3bab5f3ed1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.593551 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 086b40036f0c4c97bb95647f91ce7939 (DEFAULT_TABLE table=default.Table_c4c451a485f24525b3dbb3f3626a01e7 [id=5225751c079f4b7cbd21bc817c91851e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.760502 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086b40036f0c4c97bb95647f91ce7939. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.593835 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 106524801a204a81b9f194321768f433 (DEFAULT_TABLE table=default.Table_11f1980aeb6a4f38a5c9eebc431ddd86 [id=a3113643104d49949c503729cb7d87a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.764532 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 106524801a204a81b9f194321768f433. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.594099 2233 tablet_service.cc:1505] Processing CreateTablet for tablet d01fa8b270e1476cb86d81a196ba94f8 (DEFAULT_TABLE table=default.Table_d9351d4a858248d8813e9df05f900010 [id=06e6f90503e344cb993a0fb798f690bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.768539 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d01fa8b270e1476cb86d81a196ba94f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.594378 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 6cbea840b88c4ef6b140a9a769fd69df (DEFAULT_TABLE table=default.Table_4996540a4e3749c6aa9bdecf10b2fae3 [id=ffe25b8112e447fa85edc84784f9c95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.609400 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 68ad3d4a2d484badb4b35884517b72f1 (DEFAULT_TABLE table=default.Table_890ffb18fe5a49f1938a0fca64189907 [id=5e42165433a248a9bff99b74846d72e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.780344 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68ad3d4a2d484badb4b35884517b72f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.619913 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 4ec5b59f3b554525a0e1a7d57ec2962f (DEFAULT_TABLE table=default.Table_51306871ca71474db5412b80acf6f5e7 [id=10bab83b438b4584ab88a7f1241809c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.783771 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec5b59f3b554525a0e1a7d57ec2962f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.791767 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 9a08e8b7eb1949408f2151f2d7257e9d (table default.Table_c02fc4c8f1994cabaee4a6aaf1874bdd [id=43cb7f538f824ae38059645230cc7e94]) successfully deleted
W20251026 02:56:34.792029 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet b1ca5b8379c94deab1130bfe8bdc87b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1ca5b8379c94deab1130bfe8bdc87b0 already in progress: creating tablet
I20251026 02:56:34.792626 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet c570e5f9a93c45bda32ec92c68794714 (table default.Table_0a8522a9059f4189b8cb07ff7409bd3a [id=3d118d55d42047bdbc0b8fb944b3a190]) successfully deleted
I20251026 02:56:34.792728 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cbea840b88c4ef6b140a9a769fd69df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.623821 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 31cae8e4448549fd827f2bf836d9634d (DEFAULT_TABLE table=default.Table_c4160ae9bbe94aa8a45dbf23888f7aaf [id=bc99d7ca957d43589c58153b199f45ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.796607 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31cae8e4448549fd827f2bf836d9634d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.797988 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_84ccb1774a294a1b823ffb7ee68e621f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:34.798632 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84ccb1774a294a1b823ffb7ee68e621f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.626781 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 6699b50d87964f6a89c365171253ad25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:34.800186 2226 tablet_service.cc:1505] Processing CreateTablet for tablet 8d7c172802db489788bdc75231aa47d6 (DEFAULT_TABLE table=default.Table_5b564a01de564f7aa92845f3d962af11 [id=fa52e0b6347c47ef817d43b599861c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.800359 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f78068ab6b104363a2ed09be81de7ea2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:34.800657 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d7c172802db489788bdc75231aa47d6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:34.800879 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f78068ab6b104363a2ed09be81de7ea2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.803210 2195 leader_election.cc:304] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:34.629784 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet 5824831b8e184daf8b40543686c06467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:34.804245 3252 raft_consensus.cc:2804] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:34.804352 3252 raft_consensus.cc:697] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:34.804548 3252 consensus_queue.cc:237] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.652452 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 848e414f6e134761818039f7e98500ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:34.805310 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 7e1a45412db1449a99252112d8110a17 (DEFAULT_TABLE table=default.Table_fa3b58cf82f242cebd20cc1ea9b2133d [id=4a727b205b89448fb97b6bde0c3dd210]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.805662 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e1a45412db1449a99252112d8110a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.804545 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 7f5cd37c87f34a40bf05829af6de0473 (DEFAULT_TABLE table=default.Table_d450815506d645aab06dee257de7ea3e [id=f04a81a5e5ec403d8533855c82814d07]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.808669 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f5cd37c87f34a40bf05829af6de0473. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.694438 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 (DEFAULT_TABLE table=default.Table_61f1d10f16b1467aaab14fe4fbaef1bc [id=216da41d45b24046b44678c1ff34c17e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.812548 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.812920 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b42f8c686f234486a22954c2d04a51b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:34.813635 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b42f8c686f234486a22954c2d04a51b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.815531 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_55f34abf69314dfab04ededf15472ee8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:34.815989 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55f34abf69314dfab04ededf15472ee8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.694825 2227 tablet_service.cc:1505] Processing CreateTablet for tablet 1219e8bdfde9444f9bfdf1d626389e88 (DEFAULT_TABLE table=default.Table_aca29902287d4d8fa34e85821bfa93db [id=a7b8f1c1554d44e1b2aceff82f0771f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.816517 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1219e8bdfde9444f9bfdf1d626389e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.695008 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ddc7972f29a429ea51f6c6c1a074baf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:34.820657 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 0efa5bb6b73e493fa53f549522aadb33 (DEFAULT_TABLE table=default.Table_5322450167b9484c9e1448d711dc9702 [id=28ea5b74b9f848ad9e1531de21ab2cd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.821326 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0efa5bb6b73e493fa53f549522aadb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.824252 3028 raft_consensus.cc:697] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:34.824571 3028 consensus_queue.cc:237] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.828553 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7b456a8685d84372a3b4710e3ad99da5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:34.829231 2195 leader_election.cc:304] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
W20251026 02:56:34.829488 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b456a8685d84372a3b4710e3ad99da5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.829567 2962 raft_consensus.cc:2804] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:34.829663 2962 raft_consensus.cc:697] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:34.829865 2962 consensus_queue.cc:237] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.832633 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_bc49906123d84d55a0849db64bb7c25a" } new_table_name: "default.Table_1ae4398bf61e405697bf484105183722" modify_external_catalogs: true
W20251026 02:56:34.845990 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 144b957c69e64085a6f78050dea426b4 consensus state has no leader
I20251026 02:56:34.846985 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_aca29902287d4d8fa34e85821bfa93db" } modify_external_catalogs: true
I20251026 02:56:34.847388 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_aca29902287d4d8fa34e85821bfa93db" } modify_external_catalogs: true
I20251026 02:56:34.848979 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_76c2c5fa88984a46ac25402fa3d04654" } modify_external_catalogs: true
I20251026 02:56:34.849279 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_76c2c5fa88984a46ac25402fa3d04654" } modify_external_catalogs: true
I20251026 02:56:34.851038 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 7c82a43e97cd4f0a9cb24c598238d849 (DEFAULT_TABLE table=default.Table_d48a43604ff140f0b7103cc9fb517a5c [id=1da8a6913ca44ef7b03a7d87e41e1e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.851656 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8cfa87d9be80442b941db99ddc6021d2" } modify_external_catalogs: true
I20251026 02:56:34.851874 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8cfa87d9be80442b941db99ddc6021d2" } modify_external_catalogs: true
I20251026 02:56:34.852710 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_44853e936f824114a26a5c01ae8543d3" } modify_external_catalogs: true
I20251026 02:56:34.852878 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_44853e936f824114a26a5c01ae8543d3" } modify_external_catalogs: true
I20251026 02:56:34.853600 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_91f57fb05381498c89f99d19bdc7cd9b" } modify_external_catalogs: true
I20251026 02:56:34.853777 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_91f57fb05381498c89f99d19bdc7cd9b" } modify_external_catalogs: true
I20251026 02:56:34.854472 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6a9abf3b541f42cda3bbf2f44d86c970" } new_table_name: "default.Table_42e3536941484e83a992e13086dc316e" modify_external_catalogs: true
I20251026 02:56:34.857662 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_02d48c3527e64c1e9f77b7a9d89ab372"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:34.858508 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02d48c3527e64c1e9f77b7a9d89ab372 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.859725 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_55b76386af9a4934813a5c92774f5697"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:34.860318 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_da48008b6d1a4e0b8dd4f80c2bfad18a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:34.860929 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da48008b6d1a4e0b8dd4f80c2bfad18a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:34.863430 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55b76386af9a4934813a5c92774f5697 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:34.864254 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f3ae3a377af46449c3faca339f86053 consensus state has no leader
I20251026 02:56:34.894665 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet bc4ca4423e3a4e0c96846babec931fec
I20251026 02:56:34.895731 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a74d6a4e041347578b0976a8b54bf22c
I20251026 02:56:34.897020 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 657e07d0b17c44d4b78b97210f17ce40
I20251026 02:56:34.898171 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3efafaac6c634d088396d56c8debd990
W20251026 02:56:34.908385 1647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.60:33729, user_credentials={real_user=slave}} blocked reactor thread for 52059.4us
I20251026 02:56:34.857954 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 8ae2fa984c454a199e12eac3c0ed6684 (DEFAULT_TABLE table=default.Table_c76a8a4746f9463eb166591b7c7b5713 [id=8259cee436bd46ed9cc0c0a3598044ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.868568 3245 consensus_queue.cc:1048] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:34.912675 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae2fa984c454a199e12eac3c0ed6684. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.875342 2239 tablet_service.cc:1505] Processing CreateTablet for tablet bc34ad9deb6d4fcb87d0c0fdfee17bbf (DEFAULT_TABLE table=default.Table_6a273a7631fc410e8ba77d604a7e1c66 [id=1af4863cbdeb4f88a616a450022541b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.878505 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 1360fe4278f24f31b4ea3eb28d5abf5d (DEFAULT_TABLE table=default.Table_53a2c0226802442a8954f3903443b574 [id=881224ae789e4daab936faafe4765729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.885572 2226 tablet_service.cc:1505] Processing CreateTablet for tablet 9a5670a9e10c4a9db64881e7e14e839e (DEFAULT_TABLE table=default.Table_e250def75d2042ee89b9ce3bd7ed34ab [id=12c3c591a85f4e1187bebde0eb667c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.888044 2236 tablet_service.cc:1505] Processing CreateTablet for tablet 1c740ab6481e4481b95878c3a185129d (DEFAULT_TABLE table=default.Table_b56f9f35731e4bc4be24623e8e79fe5e [id=81816130a83c4671856506639021977f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.889823 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet c09153fbfe18434393c188058a3461d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:34.900533 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 5f8d8c4fbd474517961ea831806e6503 (DEFAULT_TABLE table=default.Table_2d168c00cb884360bdb2475bd391f576 [id=65b5770c4b9243b892bfab963a439bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.903602 2225 tablet_service.cc:1552] Processing DeleteTablet for tablet efd751ee4b1d433198dcd00c081f648d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:34.914775 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 244c03a9293242a48077bdaf90a47865 (DEFAULT_TABLE table=default.Table_7ece114e4cfd4133871448f9ea3ce9e2 [id=c8a08f2146b240698d60f43f1bc39a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.915347 2233 tablet_service.cc:1505] Processing CreateTablet for tablet d85848daf97044e980060490a145400c (DEFAULT_TABLE table=default.Table_067a40b223504137ad333312ad69073a [id=c5e5c1368be1483d979623e3bff6a485]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.916617 2227 tablet_service.cc:1505] Processing CreateTablet for tablet bdfb4ee8b9c843d19929f9b31ba1a235 (DEFAULT_TABLE table=default.Table_cf80bd3545d84f44bceb3b4e7cd5961b [id=b5213af18efa4a94a8776ab87e3e8314]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.917695 2236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c740ab6481e4481b95878c3a185129d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.918154 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1360fe4278f24f31b4ea3eb28d5abf5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.918470 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 4be3e9d5b07a48999497ed54a3d035fe (DEFAULT_TABLE table=default.Table_bc0096c9da9c4603a0d2ea3538682695 [id=c89fbfa5c47d4bcca61df44679baddd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.919099 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 91f81d156ff44decb63a4c0f5eacdac9 (DEFAULT_TABLE table=default.Table_2d0444a3af524d67ac524f71f4d70f11 [id=a5a144359e114343be80ece376d869b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.920460 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 0cdd8ce0b02c498ab9110b74de285705 (DEFAULT_TABLE table=default.Table_2704c1510c6442c1b922a5ed64bb7084 [id=032fbcc1b2a643e784183d0505032f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.905881 2230 tablet_service.cc:1505] Processing CreateTablet for tablet ce11081bf85f44e8bf1f92ce0f4bf152 (DEFAULT_TABLE table=default.Table_26257eca63ce4baaa75beae2e9db1340 [id=fbc97817a9584474abc0b2ab2052abb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.906426 2240 tablet_service.cc:1505] Processing CreateTablet for tablet cef68c4d9a3e4a85acdab446bd6082ce (DEFAULT_TABLE table=default.Table_43ea6a50a51e416da1649a35e72a7283 [id=1dbfa20801f4400d9edd25403711e212]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.924942 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 244c03a9293242a48077bdaf90a47865. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.925343 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a5670a9e10c4a9db64881e7e14e839e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.925594 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 15f9efe5bf8242a3818f05b6c8baea86 (DEFAULT_TABLE table=default.Table_c25e8ab335f34833a18f68c6585fd7a2 [id=6f7784e82aa24d889b552dcbce16003a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.926870 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfb4ee8b9c843d19929f9b31ba1a235. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.930038 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d85848daf97044e980060490a145400c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.930675 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f8d8c4fbd474517961ea831806e6503. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.930932 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be3e9d5b07a48999497ed54a3d035fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.931216 2238 tablet_service.cc:1505] Processing CreateTablet for tablet 887dc91626664f23b39cf70a6bb9381e (DEFAULT_TABLE table=default.Table_36597a9424aa42128d73327bd78be6fd [id=28d622f785bd431d86ab95a239267a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.910449 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 2e0b929f603b4c25b5b971ee9db683e8 (DEFAULT_TABLE table=default.Table_81e9f54ed3934b6b80f7c10d0be54d9a [id=e2f6dfa3686b41ddbc9f3b0205c7f05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.941036 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15f9efe5bf8242a3818f05b6c8baea86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.941859 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91f81d156ff44decb63a4c0f5eacdac9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:34.944871 2194 consensus_peers.cc:597] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:34.945180 2194 consensus_peers.cc:597] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:34.945688 2194 consensus_peers.cc:597] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:34.946370 2194 leader_election.cc:343] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:34.946900 2194 leader_election.cc:343] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:34.947525 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc34ad9deb6d4fcb87d0c0fdfee17bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.948765 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cdd8ce0b02c498ab9110b74de285705. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.949354 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce11081bf85f44e8bf1f92ce0f4bf152. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.949648 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef68c4d9a3e4a85acdab446bd6082ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.951990 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e0b929f603b4c25b5b971ee9db683e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.777995 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "046a34f9408b406ba08471421f422064" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:34.959219 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae95654d129441e08fcb3f3b355ffc22" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:34.959841 2377 log.cc:1199] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/a08efd37968846d2958e54831488235c
I20251026 02:56:34.972709 1978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.150.2:44453 (request call id 117) took 1476 ms. Trace:
I20251026 02:56:34.973006 2376 ts_tablet_manager.cc:1929] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:34.973136 2376 log.cc:1199] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/a111848706b743c9beb2fbd21f59d7f3
I20251026 02:56:34.973484 2376 ts_tablet_manager.cc:1950] T a111848706b743c9beb2fbd21f59d7f3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:34.973804 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 471) took 3791 ms. Trace:
I20251026 02:56:34.973881 2376 rpcz_store.cc:276] 1026 02:56:31.182421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.182499 (+ 78us) service_pool.cc:225] Handling call
1026 02:56:34.381865 (+3199366us) ts_tablet_manager.cc:1151] Deleting tablet a111848706b743c9beb2fbd21f59d7f3
1026 02:56:34.381874 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:34.538216 (+156342us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:34.972898 (+434682us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:34.973787 (+ 889us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3199114}
I20251026 02:56:34.974261 2376 tablet_replica.cc:333] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:34.974460 2376 raft_consensus.cc:2243] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:34.974617 2376 raft_consensus.cc:2272] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:34.975207 2376 ts_tablet_manager.cc:1916] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:34.714094 3206 tablet.cc:1722] T 740f7c7f186f45ceaa4c40e76419e3a1 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251026 02:56:34.712819 2120 raft_consensus.cc:2468] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:34.977705 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 504) took 3585 ms. Trace:
I20251026 02:56:34.977787 2195 leader_election.cc:304] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:34.977784 2372 rpcz_store.cc:276] 1026 02:56:31.391939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.392007 (+ 68us) service_pool.cc:225] Handling call
1026 02:56:34.977553 (+3585546us) ts_tablet_manager.cc:1151] Deleting tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5
1026 02:56:34.977565 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:34.977688 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3585264}
I20251026 02:56:34.978111 3245 raft_consensus.cc:2804] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:34.978228 3245 raft_consensus.cc:697] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:34.978768 2372 tablet_replica.cc:333] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:34.978946 2372 raft_consensus.cc:2243] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:34.979095 2372 raft_consensus.cc:2272] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:34.979604 2372 ts_tablet_manager.cc:1916] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:34.983098 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1219e8bdfde9444f9bfdf1d626389e88
W20251026 02:56:34.764246 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 113535us
W20251026 02:56:34.985329 2194 leader_election.cc:343] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:34.822175 2375 tablet_bootstrap.cc:492] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:34.709501 2116 raft_consensus.cc:2468] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:34.953739 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 8ae2fa984c454a199e12eac3c0ed6684 (DEFAULT_TABLE table=default.Table_c76a8a4746f9463eb166591b7c7b5713 [id=8259cee436bd46ed9cc0c0a3598044ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.988471 2375 ts_tablet_manager.cc:1403] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.602s user 0.003s sys 0.007s
I20251026 02:56:34.988761 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae2fa984c454a199e12eac3c0ed6684. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.954427 2095 tablet_service.cc:1505] Processing CreateTablet for tablet bc34ad9deb6d4fcb87d0c0fdfee17bbf (DEFAULT_TABLE table=default.Table_6a273a7631fc410e8ba77d604a7e1c66 [id=1af4863cbdeb4f88a616a450022541b2]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:34.988801 2053 leader_election.cc:343] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:34.954963 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 1360fe4278f24f31b4ea3eb28d5abf5d (DEFAULT_TABLE table=default.Table_53a2c0226802442a8954f3903443b574 [id=881224ae789e4daab936faafe4765729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.955046 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet c09153fbfe18434393c188058a3461d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:34.955363 2098 tablet_service.cc:1505] Processing CreateTablet for tablet ce11081bf85f44e8bf1f92ce0f4bf152 (DEFAULT_TABLE table=default.Table_26257eca63ce4baaa75beae2e9db1340 [id=fbc97817a9584474abc0b2ab2052abb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.955469 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 4be3e9d5b07a48999497ed54a3d035fe (DEFAULT_TABLE table=default.Table_bc0096c9da9c4603a0d2ea3538682695 [id=c89fbfa5c47d4bcca61df44679baddd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.955689 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 9a5670a9e10c4a9db64881e7e14e839e (DEFAULT_TABLE table=default.Table_e250def75d2042ee89b9ce3bd7ed34ab [id=12c3c591a85f4e1187bebde0eb667c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.989106 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc34ad9deb6d4fcb87d0c0fdfee17bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.955859 2089 tablet_service.cc:1505] Processing CreateTablet for tablet 8d7c172802db489788bdc75231aa47d6 (DEFAULT_TABLE table=default.Table_5b564a01de564f7aa92845f3d962af11 [id=fa52e0b6347c47ef817d43b599861c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.956007 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 1c740ab6481e4481b95878c3a185129d (DEFAULT_TABLE table=default.Table_b56f9f35731e4bc4be24623e8e79fe5e [id=81816130a83c4671856506639021977f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.989360 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet ae95654d129441e08fcb3f3b355ffc22 with delete_type TABLET_DATA_DELETED (replaced by tablet c2d6691bb4cf4a8292b0c082e54a91a5) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:34.989729 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 244c03a9293242a48077bdaf90a47865 (DEFAULT_TABLE table=default.Table_7ece114e4cfd4133871448f9ea3ce9e2 [id=c8a08f2146b240698d60f43f1bc39a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.989732 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5221808b553f44bc82646e59a4f6881e" } new_table_name: "default.Table_b32c2e53d837476d9a0d5ae8ea1843d0" modify_external_catalogs: true
I20251026 02:56:34.956367 2088 tablet_service.cc:1505] Processing CreateTablet for tablet cef68c4d9a3e4a85acdab446bd6082ce (DEFAULT_TABLE table=default.Table_43ea6a50a51e416da1649a35e72a7283 [id=1dbfa20801f4400d9edd25403711e212]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.956377 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 5f8d8c4fbd474517961ea831806e6503 (DEFAULT_TABLE table=default.Table_2d168c00cb884360bdb2475bd391f576 [id=65b5770c4b9243b892bfab963a439bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.956785 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 91f81d156ff44decb63a4c0f5eacdac9 (DEFAULT_TABLE table=default.Table_2d0444a3af524d67ac524f71f4d70f11 [id=a5a144359e114343be80ece376d869b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.957188 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 2e0b929f603b4c25b5b971ee9db683e8 (DEFAULT_TABLE table=default.Table_81e9f54ed3934b6b80f7c10d0be54d9a [id=e2f6dfa3686b41ddbc9f3b0205c7f05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.957479 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 741792cf96164695b03de3fb86b12602 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:34.957720 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet d65b0d79e2e54af9810022b683c3288b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:34.957962 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b7d85a2941545ceaa16b22e8328b4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:34.990770 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c82a43e97cd4f0a9cb24c598238d849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.991339 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f78068ab6b104363a2ed09be81de7ea2" } new_table_name: "default.Table_6193b9cb563841bcbf42d20cf7a21f59" modify_external_catalogs: true
I20251026 02:56:34.992249 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 887dc91626664f23b39cf70a6bb9381e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:34.994910 2377 ts_tablet_manager.cc:1950] T a08efd37968846d2958e54831488235c P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:34.996739 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6722958282e540edaae2edf477b15a85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:34.997253 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6722958282e540edaae2edf477b15a85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.998559 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fbd4d396350c48129da0be37e6c3bfc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:34.998998 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbd4d396350c48129da0be37e6c3bfc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:34.994746 2195 leader_election.cc:304] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:35.000316 3144 raft_consensus.cc:2804] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:35.000463 3144 raft_consensus.cc:697] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:35.000536 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b51c545ae1d248b293e87daf463aaf8f" } new_table_name: "default.Table_381e082f8f9b4d589c346ba1879166bd" modify_external_catalogs: true
I20251026 02:56:35.000691 3144 consensus_queue.cc:237] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:34.998654 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741792cf96164695b03de3fb86b12602" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:34.978485 3245 consensus_queue.cc:237] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:35.044795 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 1c740ab6481e4481b95878c3a185129d (DEFAULT_TABLE table=default.Table_b56f9f35731e4bc4be24623e8e79fe5e [id=81816130a83c4671856506639021977f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.045255 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c740ab6481e4481b95878c3a185129d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:35.049373 1647 consensus_peers.cc:597] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 -> Peer 751b719b75ce4bc2a88a19091230ecb7 (127.1.150.60:33729): Couldn't send request to peer 751b719b75ce4bc2a88a19091230ecb7. Status: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251026 02:56:35.061159 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 4be3e9d5b07a48999497ed54a3d035fe (DEFAULT_TABLE table=default.Table_bc0096c9da9c4603a0d2ea3538682695 [id=c89fbfa5c47d4bcca61df44679baddd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.063414 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet c09153fbfe18434393c188058a3461d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:35.068804 3187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 02:56:35.069536 3187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 02:56:35.069696 3187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251026 02:56:35.071916 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 8d7c172802db489788bdc75231aa47d6 (DEFAULT_TABLE table=default.Table_5b564a01de564f7aa92845f3d962af11 [id=fa52e0b6347c47ef817d43b599861c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.072475 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d7c172802db489788bdc75231aa47d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.079813 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be3e9d5b07a48999497ed54a3d035fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.081018 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 5f8d8c4fbd474517961ea831806e6503 (DEFAULT_TABLE table=default.Table_2d168c00cb884360bdb2475bd391f576 [id=65b5770c4b9243b892bfab963a439bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.081606 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f8d8c4fbd474517961ea831806e6503. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.074766 1962 tablet_service.cc:1505] Processing CreateTablet for tablet cef68c4d9a3e4a85acdab446bd6082ce (DEFAULT_TABLE table=default.Table_43ea6a50a51e416da1649a35e72a7283 [id=1dbfa20801f4400d9edd25403711e212]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.082001 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef68c4d9a3e4a85acdab446bd6082ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:35.089133 3187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 02:56:35.089354 3187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 02:56:35.089558 3187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 02:56:35.089679 3187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 02:56:35.117447 1947 tablet_service.cc:1505] Processing CreateTablet for tablet bdfb4ee8b9c843d19929f9b31ba1a235 (DEFAULT_TABLE table=default.Table_cf80bd3545d84f44bceb3b4e7cd5961b [id=b5213af18efa4a94a8776ab87e3e8314]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.117918 3287 raft_consensus.cc:493] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:35.118763 1981 raft_consensus.cc:3060] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:35.122190 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet a111848706b743c9beb2fbd21f59d7f3 (table default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]) successfully deleted
I20251026 02:56:35.125878 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 887dc91626664f23b39cf70a6bb9381e (DEFAULT_TABLE table=default.Table_36597a9424aa42128d73327bd78be6fd [id=28d622f785bd431d86ab95a239267a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.126307 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 887dc91626664f23b39cf70a6bb9381e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.126502 1950 tablet_service.cc:1505] Processing CreateTablet for tablet 2e0b929f603b4c25b5b971ee9db683e8 (DEFAULT_TABLE table=default.Table_81e9f54ed3934b6b80f7c10d0be54d9a [id=e2f6dfa3686b41ddbc9f3b0205c7f05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.126793 1950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e0b929f603b4c25b5b971ee9db683e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.127653 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 244c03a9293242a48077bdaf90a47865 (DEFAULT_TABLE table=default.Table_7ece114e4cfd4133871448f9ea3ce9e2 [id=c8a08f2146b240698d60f43f1bc39a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.127943 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 244c03a9293242a48077bdaf90a47865. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.129887 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfb4ee8b9c843d19929f9b31ba1a235. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.132354 3187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.150.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.150.60:33729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.150.60
--webserver_port=39671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 26 Oct 2025 02:43:21 UTC on 24a791456cd2
build id 8720
ASAN enabled
I20251026 02:56:35.136631 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 0cdd8ce0b02c498ab9110b74de285705 (DEFAULT_TABLE table=default.Table_2704c1510c6442c1b922a5ed64bb7084 [id=032fbcc1b2a643e784183d0505032f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.137153 3250 tablet.cc:1722] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:35.137274 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cdd8ce0b02c498ab9110b74de285705. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.137594 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 15f9efe5bf8242a3818f05b6c8baea86 (DEFAULT_TABLE table=default.Table_c25e8ab335f34833a18f68c6585fd7a2 [id=6f7784e82aa24d889b552dcbce16003a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.137920 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15f9efe5bf8242a3818f05b6c8baea86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.138582 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 91f81d156ff44decb63a4c0f5eacdac9 (DEFAULT_TABLE table=default.Table_2d0444a3af524d67ac524f71f4d70f11 [id=a5a144359e114343be80ece376d869b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.138880 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91f81d156ff44decb63a4c0f5eacdac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.140476 3187 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 02:56:35.145753 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 2f197526e7d34f849dbb4f28d8d414cf (DEFAULT_TABLE table=default.Table_b906304b7fc9457f98068c196bdea574 [id=fa7ef54b9b634a779bd5c6d450843d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.153007 3187 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 02:56:35.153681 1955 tablet_service.cc:1505] Processing CreateTablet for tablet b8cad7adae7e43469ceadbedeee53564 (DEFAULT_TABLE table=default.Table_0198232bf3d14a08b9ef2331469c256c [id=b89d23a3925b4d9092e79a434d6ca385]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:34.988893 2053 leader_election.cc:304] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:35.161703 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1360fe4278f24f31b4ea3eb28d5abf5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:35.161784 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 172980us
W20251026 02:56:35.162179 2053 leader_election.cc:343] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:35.162274 2053 leader_election.cc:304] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:35.162449 2092 tablet_service.cc:1505] Processing CreateTablet for tablet bdfb4ee8b9c843d19929f9b31ba1a235 (DEFAULT_TABLE table=default.Table_cf80bd3545d84f44bceb3b4e7cd5961b [id=b5213af18efa4a94a8776ab87e3e8314]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:35.162726 2053 leader_election.cc:343] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:35.162878 2053 leader_election.cc:343] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:35.163098 2053 leader_election.cc:343] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:35.163230 2053 leader_election.cc:343] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:35.163259 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce11081bf85f44e8bf1f92ce0f4bf152. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.164422 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a5670a9e10c4a9db64881e7e14e839e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.168457 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c740ab6481e4481b95878c3a185129d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.172586 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 244c03a9293242a48077bdaf90a47865. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:35.018383 2194 leader_election.cc:343] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:35.089917 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 64faa109f006414bb955ed483ecc189e (DEFAULT_TABLE table=default.Table_18ed9c54e04542528f377fd10c75cf9c [id=4861e60511684b91b02ca67675862932]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:35.174758 2194 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 156381us
I20251026 02:56:35.175004 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64faa109f006414bb955ed483ecc189e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:35.175148 2194 leader_election.cc:343] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:35.175551 2194 leader_election.cc:343] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:35.175801 2194 leader_election.cc:343] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:35.176119 2194 consensus_peers.cc:597] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:35.176694 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f8d8c4fbd474517961ea831806e6503. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.178745 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet a74d6a4e041347578b0976a8b54bf22c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:35.178901 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 657e07d0b17c44d4b78b97210f17ce40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:35.179051 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 3efafaac6c634d088396d56c8debd990 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:35.179214 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet 1219e8bdfde9444f9bfdf1d626389e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:35.102202 2224 tablet_service.cc:1505] Processing CreateTablet for tablet b8cad7adae7e43469ceadbedeee53564 (DEFAULT_TABLE table=default.Table_0198232bf3d14a08b9ef2331469c256c [id=b89d23a3925b4d9092e79a434d6ca385]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.179755 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8cad7adae7e43469ceadbedeee53564. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.111742 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 88b0cca095514614817346d6eb9014f5 (DEFAULT_TABLE table=default.Table_7a84416153d6424eb94a56932605f7ad [id=8e6c217882d441c7a218e38985b0a157]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.180862 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88b0cca095514614817346d6eb9014f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.181722 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e0b929f603b4c25b5b971ee9db683e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.045876 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 449) took 5796 ms. Trace:
I20251026 02:56:35.158563 1952 tablet_service.cc:1505] Processing CreateTablet for tablet 88b0cca095514614817346d6eb9014f5 (DEFAULT_TABLE table=default.Table_7a84416153d6424eb94a56932605f7ad [id=8e6c217882d441c7a218e38985b0a157]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.180253 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f197526e7d34f849dbb4f28d8d414cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.184233 2377 rpcz_store.cc:276] 1026 02:56:29.248973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.249028 (+ 55us) service_pool.cc:225] Handling call
1026 02:56:34.557702 (+5308674us) ts_tablet_manager.cc:1151] Deleting tablet a08efd37968846d2958e54831488235c
1026 02:56:34.557711 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:34.565914 (+ 8203us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:34.570372 (+ 4458us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.045848 (+475476us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5288104}
I20251026 02:56:35.184499 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef68c4d9a3e4a85acdab446bd6082ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.184731 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8cad7adae7e43469ceadbedeee53564. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.184705 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 453) took 5851 ms. Trace:
I20251026 02:56:35.184792 2377 rpcz_store.cc:276] 1026 02:56:29.332785 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.332844 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:35.184559 (+5851715us) ts_tablet_manager.cc:1151] Deleting tablet ce922040298d4952af13652eeda7a234
1026 02:56:35.184569 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.184686 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5851442}
I20251026 02:56:35.184819 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet a08efd37968846d2958e54831488235c (table default.Table_80ca368a0ade4ac2b090b31b2ab56f12 [id=49dd7ccfa92946bebeaa4c178138f75a]) successfully deleted
I20251026 02:56:35.185125 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 454) took 5811 ms. Trace:
I20251026 02:56:35.185189 2377 rpcz_store.cc:276] 1026 02:56:29.373998 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.374055 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:35.185019 (+5810964us) ts_tablet_manager.cc:1151] Deleting tablet f2d97084b5914350afbfda5c2a34bf9a
1026 02:56:35.185028 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.185110 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5810705}
W20251026 02:56:35.185371 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ce922040298d4952af13652eeda7a234 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce922040298d4952af13652eeda7a234 already in progress: creating tablet
I20251026 02:56:35.185482 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 464) took 5396 ms. Trace:
I20251026 02:56:35.185535 2377 rpcz_store.cc:276] 1026 02:56:29.789207 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.791350 (+ 2143us) service_pool.cc:225] Handling call
1026 02:56:35.185404 (+5394054us) ts_tablet_manager.cc:1151] Deleting tablet 0e05eada8742471f87d73f5d960817c0
1026 02:56:35.185412 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.185470 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5393790}
W20251026 02:56:35.185747 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet f2d97084b5914350afbfda5c2a34bf9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2d97084b5914350afbfda5c2a34bf9a already in progress: creating tablet
I20251026 02:56:35.185781 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 466) took 5379 ms. Trace:
I20251026 02:56:35.185833 2377 rpcz_store.cc:276] 1026 02:56:29.805996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.806987 (+ 991us) service_pool.cc:225] Handling call
1026 02:56:35.185707 (+5378720us) ts_tablet_manager.cc:1151] Deleting tablet 79c18ebf56e045b6a52fae8871a9107f
1026 02:56:35.185713 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.185767 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5378409}
I20251026 02:56:35.186087 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 468) took 5357 ms. Trace:
W20251026 02:56:35.186118 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0e05eada8742471f87d73f5d960817c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e05eada8742471f87d73f5d960817c0 already in progress: creating tablet
I20251026 02:56:35.186141 2377 rpcz_store.cc:276] 1026 02:56:29.829133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:29.829195 (+ 62us) service_pool.cc:225] Handling call
1026 02:56:35.185990 (+5356795us) ts_tablet_manager.cc:1151] Deleting tablet bfa68f2c942c43e2a02e9d333bdfa2a2
1026 02:56:35.185997 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.186076 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5356527}
W20251026 02:56:35.186342 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 79c18ebf56e045b6a52fae8871a9107f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79c18ebf56e045b6a52fae8871a9107f already in progress: creating tablet
I20251026 02:56:35.186400 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 470) took 5174 ms. Trace:
I20251026 02:56:35.186451 2377 rpcz_store.cc:276] 1026 02:56:30.011872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.011941 (+ 69us) service_pool.cc:225] Handling call
1026 02:56:35.186328 (+5174387us) ts_tablet_manager.cc:1151] Deleting tablet e89a98fcfdbb4011876b0ebc2af91f67
1026 02:56:35.186335 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.186389 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5174112}
I20251026 02:56:35.186710 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 471) took 5112 ms. Trace:
I20251026 02:56:35.186759 2377 rpcz_store.cc:276] 1026 02:56:30.074164 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.074908 (+ 744us) service_pool.cc:225] Handling call
1026 02:56:35.186597 (+5111689us) ts_tablet_manager.cc:1151] Deleting tablet 95c9fbae62db4850b0f02a2443a3432b
1026 02:56:35.186604 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.186698 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5107274}
W20251026 02:56:35.187000 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet e89a98fcfdbb4011876b0ebc2af91f67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e89a98fcfdbb4011876b0ebc2af91f67 already in progress: creating tablet
I20251026 02:56:35.187047 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 472) took 5014 ms. Trace:
I20251026 02:56:35.187114 2377 rpcz_store.cc:276] 1026 02:56:30.172381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.172443 (+ 62us) service_pool.cc:225] Handling call
1026 02:56:35.186958 (+5014515us) ts_tablet_manager.cc:1151] Deleting tablet c1dd78f088f84a509c761c934c039016
1026 02:56:35.186965 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.187032 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5014252}
W20251026 02:56:35.187356 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 95c9fbae62db4850b0f02a2443a3432b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95c9fbae62db4850b0f02a2443a3432b already in progress: creating tablet
I20251026 02:56:35.187403 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 473) took 4923 ms. Trace:
I20251026 02:56:35.187458 2377 rpcz_store.cc:276] 1026 02:56:30.263544 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.263791 (+ 247us) service_pool.cc:225] Handling call
1026 02:56:35.187324 (+4923533us) ts_tablet_manager.cc:1151] Deleting tablet 99d318aa0d1847c58167bf0af9e75aa2
1026 02:56:35.187331 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.187392 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4923283}
I20251026 02:56:35.187641 2377 tablet_replica.cc:333] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539: stopping tablet replica
W20251026 02:56:35.187737 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet c1dd78f088f84a509c761c934c039016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1dd78f088f84a509c761c934c039016 already in progress: creating tablet
I20251026 02:56:35.187795 2377 raft_consensus.cc:2243] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:35.187937 2377 raft_consensus.cc:2272] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:35.187996 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 99d318aa0d1847c58167bf0af9e75aa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99d318aa0d1847c58167bf0af9e75aa2 already in progress: creating tablet
I20251026 02:56:35.188014 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 64faa109f006414bb955ed483ecc189e (DEFAULT_TABLE table=default.Table_18ed9c54e04542528f377fd10c75cf9c [id=4861e60511684b91b02ca67675862932]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.117336 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 77ecdcbe25e24267bf1066a1baad804a (DEFAULT_TABLE table=default.Table_1abe312e689149299920c58ec6f1a35d [id=e652cb77eee74028b76a98c922b82f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.189052 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77ecdcbe25e24267bf1066a1baad804a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.162448 3006 raft_consensus.cc:2749] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:35.190526 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 77ecdcbe25e24267bf1066a1baad804a (DEFAULT_TABLE table=default.Table_1abe312e689149299920c58ec6f1a35d [id=e652cb77eee74028b76a98c922b82f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.191674 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88b0cca095514614817346d6eb9014f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.191938 3279 raft_consensus.cc:493] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:35.119802 2243 tablet_service.cc:1505] Processing CreateTablet for tablet e32a1f8dcad4430eb35d388bfad7d110 (DEFAULT_TABLE table=default.Table_7921be016a944a41953b751ba927dacd [id=d3c0e21fa8bd430ea09e47bb699e3704]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.192574 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32a1f8dcad4430eb35d388bfad7d110. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.193413 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91f81d156ff44decb63a4c0f5eacdac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.193711 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d7c172802db489788bdc75231aa47d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.194023 3287 raft_consensus.cc:515] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.126505 2240 tablet_service.cc:1505] Processing CreateTablet for tablet bd11b620e78c499ebaa1b35f361242c3 (DEFAULT_TABLE table=default.Table_b51c545ae1d248b293e87daf463aaf8f [id=9158d553997949bfbe37cf153f6b1ab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.196574 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd11b620e78c499ebaa1b35f361242c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.128917 2241 tablet_service.cc:1505] Processing CreateTablet for tablet b8a1244c1c49471fb7711a2ca9b89a99 (DEFAULT_TABLE table=default.Table_dcdbf613e1704183b39ba6294a0b202d [id=48076cd17ca94dd8928115777232e240]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.200799 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8a1244c1c49471fb7711a2ca9b89a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.139122 2381 tablet_bootstrap.cc:492] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:35.204447 2381 ts_tablet_manager.cc:1403] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.736s user 0.001s sys 0.009s
I20251026 02:56:35.143663 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 8aa0575af07e433c8d96acab31edb4af (DEFAULT_TABLE table=default.Table_901960ec4b4749f7b73a558f65408857 [id=232e25932feb4f6bbb665815b97899cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.204913 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa0575af07e433c8d96acab31edb4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.146303 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 803248412240468ea45131299f2bd5df (DEFAULT_TABLE table=default.Table_84ccb1774a294a1b823ffb7ee68e621f [id=b22fc534ade646fca6efe6af4e588c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.205384 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803248412240468ea45131299f2bd5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.192057 3279 raft_consensus.cc:515] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:35.206674 3246 consensus_queue.cc:1048] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:35.149642 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 8e3cdb1f81214337a7ccc7816755f58c (DEFAULT_TABLE table=default.Table_55f34abf69314dfab04ededf15472ee8 [id=497a284ac16f4f6e9fcad5fe49dcbe13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.151906 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 193e32898b164133b954409a08c298ce (DEFAULT_TABLE table=default.Table_f78068ab6b104363a2ed09be81de7ea2 [id=52c2da2abb1e4417bd60703a5f672e70]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:35.153755 2234 consensus_meta.cc:287] T 82e4bc6195484074b0da610a8d1b65bc P 696476265495492790739ec126a08539: Time spent flushing consensus metadata: real 0.524s user 0.003s sys 0.000s
I20251026 02:56:35.212867 2234 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35450 (request call id 588) took 1572 ms. Trace:
I20251026 02:56:35.214566 1949 tablet_service.cc:1505] Processing CreateTablet for tablet d85848daf97044e980060490a145400c (DEFAULT_TABLE table=default.Table_067a40b223504137ad333312ad69073a [id=c5e5c1368be1483d979623e3bff6a485]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.217110 2381 raft_consensus.cc:359] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.217269 2381 raft_consensus.cc:385] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:35.217355 2381 raft_consensus.cc:740] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:35.148075 2238 tablet_service.cc:1505] Processing CreateTablet for tablet f383f638e63040379bf749b8774fd7ed (DEFAULT_TABLE table=default.Table_b42f8c686f234486a22954c2d04a51b5 [id=c0d677d50d3d4de3a51a9aa68b12ce02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.200309 3272 raft_consensus.cc:2749] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:35.212987 2375 raft_consensus.cc:359] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:35.218371 2375 raft_consensus.cc:385] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:35.218432 2375 raft_consensus.cc:740] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:35.218760 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3cdb1f81214337a7ccc7816755f58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.218657 2375 consensus_queue.cc:260] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:35.219314 2375 ts_tablet_manager.cc:1434] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.231s user 0.001s sys 0.001s
I20251026 02:56:35.212939 2234 rpcz_store.cc:276] 1026 02:56:33.640292 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.640571 (+ 279us) service_pool.cc:225] Handling call
1026 02:56:33.641256 (+ 685us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:33.641297 (+ 41us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:34.629390 (+988093us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.212718 (+583328us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:35.212850 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 02:56:35.214506 2084 consensus_meta.cc:287] T 19016e8e11cc42c69ce44a819db9e298 P 0b6b0a6c4e144982b67980118cddba48: Time spent flushing consensus metadata: real 0.518s user 0.003s sys 0.000s
I20251026 02:56:35.219751 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f383f638e63040379bf749b8774fd7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.214787 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 0cdd8ce0b02c498ab9110b74de285705 (DEFAULT_TABLE table=default.Table_2704c1510c6442c1b922a5ed64bb7084 [id=032fbcc1b2a643e784183d0505032f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.214980 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be3e9d5b07a48999497ed54a3d035fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:35.220270 2090 consensus_meta.cc:287] T 106524801a204a81b9f194321768f433 P 0b6b0a6c4e144982b67980118cddba48: Time spent flushing consensus metadata: real 0.593s user 0.000s sys 0.003s
I20251026 02:56:35.221577 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfb4ee8b9c843d19929f9b31ba1a235. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.222076 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 15f9efe5bf8242a3818f05b6c8baea86 (DEFAULT_TABLE table=default.Table_c25e8ab335f34833a18f68c6585fd7a2 [id=6f7784e82aa24d889b552dcbce16003a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.222635 1949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d85848daf97044e980060490a145400c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.223407 2377 ts_tablet_manager.cc:1916] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:35.223508 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15f9efe5bf8242a3818f05b6c8baea86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.223734 1946 tablet_service.cc:1505] Processing CreateTablet for tablet e32a1f8dcad4430eb35d388bfad7d110 (DEFAULT_TABLE table=default.Table_7921be016a944a41953b751ba927dacd [id=d3c0e21fa8bd430ea09e47bb699e3704]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.224035 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32a1f8dcad4430eb35d388bfad7d110. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:35.205448 1915 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.010s sys 0.014s
W20251026 02:56:35.224969 1915 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.715s user 0.010s sys 0.015s
I20251026 02:56:35.225072 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 887dc91626664f23b39cf70a6bb9381e (DEFAULT_TABLE table=default.Table_36597a9424aa42128d73327bd78be6fd [id=28d622f785bd431d86ab95a239267a50]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:35.225157 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 715231us
I20251026 02:56:35.225214 3279 leader_election.cc:290] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:35.225354 1915 leader_election.cc:304] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:35.225452 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 887dc91626664f23b39cf70a6bb9381e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.225834 3279 raft_consensus.cc:2804] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:35.225925 3279 raft_consensus.cc:697] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:35.226148 3279 consensus_queue.cc:237] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:35.226326 1916 leader_election.cc:343] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9156314f98ac4ed9981f6f13fc585043
I20251026 02:56:35.226704 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cdd8ce0b02c498ab9110b74de285705. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.225732 1915 leader_election.cc:304] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:35.241519 3163 raft_consensus.cc:2749] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:35.247558 3311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:35.250453 3312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 02:56:35.260273 3315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 02:56:35.262099 3187 server_base.cc:1047] running on GCE node
I20251026 02:56:35.263239 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77ecdcbe25e24267bf1066a1baad804a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.263396 3187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 02:56:35.266551 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64faa109f006414bb955ed483ecc189e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.276124 3187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 02:56:34.972965 1978 rpcz_store.cc:276] 1026 02:56:33.496017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.496084 (+ 67us) service_pool.cc:225] Handling call
1026 02:56:33.496318 (+ 234us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 02:56:33.963322 (+467004us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 02:56:33.963328 (+ 6us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 02:56:33.963477 (+ 149us) log.cc:844] Serialized 26 byte log entry
1026 02:56:34.316821 (+353344us) raft_consensus.cc:1645] Marking committed up to 1
1026 02:56:34.725305 (+408484us) log.cc:844] Serialized 14 byte log entry
1026 02:56:34.725398 (+ 93us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 02:56:34.972589 (+247191us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 02:56:34.972603 (+ 14us) raft_consensus.cc:1685] finished
1026 02:56:34.972605 (+ 2us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 02:56:34.972679 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":272765,"prepare.run_cpu_time_us":7,"prepare.run_wall_time_us":8,"raft.queue_time_us":16387,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":112,"thread_start_us":761519,"threads_started":2,"wal-append.queue_time_us":320611}
I20251026 02:56:35.283949 3187 hybrid_clock.cc:648] HybridClock initialized: now 1761447395283817 us; error 112 us; skew 500 ppm
I20251026 02:56:35.284647 3187 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 02:56:35.295719 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 2f197526e7d34f849dbb4f28d8d414cf (DEFAULT_TABLE table=default.Table_b906304b7fc9457f98068c196bdea574 [id=fa7ef54b9b634a779bd5c6d450843d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.296743 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f197526e7d34f849dbb4f28d8d414cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.041337 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 2f197526e7d34f849dbb4f28d8d414cf (DEFAULT_TABLE table=default.Table_b906304b7fc9457f98068c196bdea574 [id=fa7ef54b9b634a779bd5c6d450843d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.173981 2227 tablet_service.cc:1552] Processing DeleteTablet for tablet bc4ca4423e3a4e0c96846babec931fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:35.311506 3287 leader_election.cc:290] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:35.311650 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2704c1510c6442c1b922a5ed64bb7084" } new_table_name: "default.Table_be5e019aee764ce0819ea54fe4398e3c" modify_external_catalogs: true
W20251026 02:56:35.313834 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 193e32898b164133b954409a08c298ce consensus state has no leader
I20251026 02:56:35.314698 3007 raft_consensus.cc:493] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:35.314821 3007 raft_consensus.cc:515] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.315013 3272 raft_consensus.cc:493] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:35.315155 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 276bd3b97c97426daf802e1f57feb265 (table default.Table_3babf04294cc48ef92df6ffe8ab43e5a [id=e91358e10bab4589a6f4e815a85a5f7d]) successfully deleted
W20251026 02:56:35.316367 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 4b5e61afe90e4bc4a2897de21e0b3919 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b5e61afe90e4bc4a2897de21e0b3919 already in progress: creating tablet
I20251026 02:56:35.316553 3006 raft_consensus.cc:493] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:35.316591 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9a345fa5edbd48d2891b8b40b09cb3f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a345fa5edbd48d2891b8b40b09cb3f8 already in progress: creating tablet
I20251026 02:56:35.316664 3006 raft_consensus.cc:515] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:35.316803 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a3af2695c41a4439bc4092b7cb1b2edf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3af2695c41a4439bc4092b7cb1b2edf already in progress: creating tablet
W20251026 02:56:35.316998 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 already in progress: creating tablet
I20251026 02:56:35.317549 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8300040c56d34cf985a2009545c1eeb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:35.317787 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c119e6aabe874f9495d15c66088c7440"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.318222 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c119e6aabe874f9495d15c66088c7440 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.319638 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_51689a3619574710a85dcec2830fc132"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:35.319815 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0207b8f2a960407fa3a85a1280bb2256"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.320230 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51689a3619574710a85dcec2830fc132 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:35.320482 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0207b8f2a960407fa3a85a1280bb2256 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:35.321576 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd11b620e78c499ebaa1b35f361242c3 consensus state has no leader
I20251026 02:56:35.321934 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a036080d1e7141269eec20b1f8e87f9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.322427 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a036080d1e7141269eec20b1f8e87f9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.322775 2375 tablet_bootstrap.cc:492] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:35.293975 2084 tablet_service.cc:1505] Processing CreateTablet for tablet f383f638e63040379bf749b8774fd7ed (DEFAULT_TABLE table=default.Table_b42f8c686f234486a22954c2d04a51b5 [id=c0d677d50d3d4de3a51a9aa68b12ce02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.323773 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d4465a7252ec4e43ac0819aab10e15d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:35.315117 3272 raft_consensus.cc:515] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.324491 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_128d37058db744249e0cd066307fbe22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.324965 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_128d37058db744249e0cd066307fbe22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.319659 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_70b9f6803b9345bc8010f0e216bff485"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.326385 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70b9f6803b9345bc8010f0e216bff485 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.328543 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5f40d3b6c1684d74a6a4577b9b3d0e85" } new_table_name: "default.Table_f9afa2d0968b401e8fc615844c100b5e" modify_external_catalogs: true
W20251026 02:56:35.328958 1916 consensus_peers.cc:597] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1a2522801364693be0bda6fa45a697e. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:35.294801 1957 tablet_service.cc:1505] Processing CreateTablet for tablet bd11b620e78c499ebaa1b35f361242c3 (DEFAULT_TABLE table=default.Table_b51c545ae1d248b293e87daf463aaf8f [id=9158d553997949bfbe37cf153f6b1ab7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:35.329818 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4465a7252ec4e43ac0819aab10e15d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.330055 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd11b620e78c499ebaa1b35f361242c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:35.318223 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8300040c56d34cf985a2009545c1eeb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.340884 2376 ts_tablet_manager.cc:1929] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:35.346423 2376 log.cc:1199] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/9e5bb0a6a9f947cf9ce119c1fc91fbf5
W20251026 02:56:35.361197 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0cdd8ce0b02c498ab9110b74de285705 consensus state has no leader
I20251026 02:56:35.365474 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ac31a485be77410ca1fafabe643f37a2" } modify_external_catalogs: true
I20251026 02:56:35.365689 2376 ts_tablet_manager.cc:1950] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:35.365725 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ac31a485be77410ca1fafabe643f37a2" } modify_external_catalogs: true
I20251026 02:56:35.366080 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 472) took 4097 ms. Trace:
I20251026 02:56:35.366169 2376 rpcz_store.cc:276] 1026 02:56:31.268809 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.268871 (+ 62us) service_pool.cc:225] Handling call
1026 02:56:34.974211 (+3705340us) ts_tablet_manager.cc:1151] Deleting tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5
1026 02:56:34.974222 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.191158 (+216936us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.340769 (+149611us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.366059 (+ 25290us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3705057}
I20251026 02:56:35.370384 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 473) took 4071 ms. Trace:
I20251026 02:56:35.371938 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 9e5bb0a6a9f947cf9ce119c1fc91fbf5 (table default.Table_25607539f64440cb8ff2c24a0a3a3bdd [id=5dbfb5d912ba433baa46bd52bbf6dbee]) successfully deleted
I20251026 02:56:35.374861 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0335b328855349de9d1c12fa3beedd0a" } modify_external_catalogs: true
I20251026 02:56:35.375092 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0335b328855349de9d1c12fa3beedd0a" } modify_external_catalogs: true
I20251026 02:56:35.370599 2376 rpcz_store.cc:276] 1026 02:56:31.299291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.299694 (+ 403us) service_pool.cc:225] Handling call
1026 02:56:35.366618 (+4066924us) ts_tablet_manager.cc:1151] Deleting tablet 276bd3b97c97426daf802e1f57feb265
1026 02:56:35.366629 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.370358 (+ 3729us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4066625}
W20251026 02:56:35.376413 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 276bd3b97c97426daf802e1f57feb265 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 276bd3b97c97426daf802e1f57feb265 already in progress: creating tablet
I20251026 02:56:35.380720 3187 webserver.cc:492] Webserver started at http://127.1.150.60:39671/ using document root <none> and password file <none>
I20251026 02:56:35.382095 3187 fs_manager.cc:362] Metadata directory not provided
I20251026 02:56:35.382249 3187 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 02:56:35.394258 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8ab273ba34e54a89910acbe0f5370372" } modify_external_catalogs: true
I20251026 02:56:35.394506 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8ab273ba34e54a89910acbe0f5370372" } modify_external_catalogs: true
I20251026 02:56:35.405100 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 8aa0575af07e433c8d96acab31edb4af (DEFAULT_TABLE table=default.Table_901960ec4b4749f7b73a558f65408857 [id=232e25932feb4f6bbb665815b97899cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.405671 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa0575af07e433c8d96acab31edb4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.408146 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_de67223546af408ead7b2d0d3979d78d" } modify_external_catalogs: true
I20251026 02:56:35.408812 1943 tablet_service.cc:1505] Processing CreateTablet for tablet f383f638e63040379bf749b8774fd7ed (DEFAULT_TABLE table=default.Table_b42f8c686f234486a22954c2d04a51b5 [id=c0d677d50d3d4de3a51a9aa68b12ce02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.409302 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f383f638e63040379bf749b8774fd7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.410127 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_de67223546af408ead7b2d0d3979d78d" } modify_external_catalogs: true
I20251026 02:56:35.412021 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f383f638e63040379bf749b8774fd7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.414227 3007 leader_election.cc:290] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:35.414695 2378 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.002s sys 0.005s
I20251026 02:56:35.414775 3187 fs_manager.cc:714] Time spent opening directory manager: real 0.021s user 0.005s sys 0.000s
W20251026 02:56:35.414850 2378 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.694s user 0.002s sys 0.005s
I20251026 02:56:35.415282 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 64faa109f006414bb955ed483ecc189e (DEFAULT_TABLE table=default.Table_18ed9c54e04542528f377fd10c75cf9c [id=4861e60511684b91b02ca67675862932]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.419878 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_065759ae67c2414ca2b73a3f68a4111c" } modify_external_catalogs: true
I20251026 02:56:35.420107 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_065759ae67c2414ca2b73a3f68a4111c" } modify_external_catalogs: true
W20251026 02:56:35.421278 2081 consensus_meta.cc:287] T a7a5020acff04c1d8b5749cdfd7dddfd P 0b6b0a6c4e144982b67980118cddba48: Time spent flushing consensus metadata: real 0.829s user 0.003s sys 0.000s
I20251026 02:56:35.422405 2081 tablet_service.cc:1505] Processing CreateTablet for tablet b8cad7adae7e43469ceadbedeee53564 (DEFAULT_TABLE table=default.Table_0198232bf3d14a08b9ef2331469c256c [id=b89d23a3925b4d9092e79a434d6ca385]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.424816 3006 leader_election.cc:290] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:35.428020 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet a74d6a4e041347578b0976a8b54bf22c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:35.430717 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_dc751f63612b4379a8d8069bc750d1f4" } new_table_name: "default.Table_11cf708e0e8243c887777e42acc25490" modify_external_catalogs: true
I20251026 02:56:35.440403 3272 leader_election.cc:290] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:35.442487 3325 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 02:56:35.443410 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64faa109f006414bb955ed483ecc189e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.448470 3187 fs_manager.cc:730] Time spent opening block manager: real 0.028s user 0.006s sys 0.001s
I20251026 02:56:35.448685 3187 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data,/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
uuid: "751b719b75ce4bc2a88a19091230ecb7"
format_stamp: "Formatted at 2025-10-26 02:56:18 on dist-test-slave-h3w0"
I20251026 02:56:35.450533 3187 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20251026 02:56:35.454102 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db94738da3554fbda64c23fde3ef6c1b consensus state has no leader
I20251026 02:56:35.459966 2375 tablet_bootstrap.cc:654] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:35.462055 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8cad7adae7e43469ceadbedeee53564. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.432672 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 803248412240468ea45131299f2bd5df (DEFAULT_TABLE table=default.Table_84ccb1774a294a1b823ffb7ee68e621f [id=b22fc534ade646fca6efe6af4e588c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.438707 1955 tablet_service.cc:1505] Processing CreateTablet for tablet b8a1244c1c49471fb7711a2ca9b89a99 (DEFAULT_TABLE table=default.Table_dcdbf613e1704183b39ba6294a0b202d [id=48076cd17ca94dd8928115777232e240]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.462958 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8a1244c1c49471fb7711a2ca9b89a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.439066 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 475) took 4011 ms. Trace:
I20251026 02:56:35.463214 2376 rpcz_store.cc:276] 1026 02:56:31.365099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.365186 (+ 87us) service_pool.cc:225] Handling call
1026 02:56:35.376822 (+4011636us) ts_tablet_manager.cc:1151] Deleting tablet 4b5e61afe90e4bc4a2897de21e0b3919
1026 02:56:35.376835 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.376960 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4011326}
W20251026 02:56:35.463889 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 4b5e61afe90e4bc4a2897de21e0b3919 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b5e61afe90e4bc4a2897de21e0b3919 already in progress: creating tablet
I20251026 02:56:35.442422 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 657e07d0b17c44d4b78b97210f17ce40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:35.465178 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 476) took 4091 ms. Trace:
I20251026 02:56:35.465327 2376 rpcz_store.cc:276] 1026 02:56:31.373819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.374362 (+ 543us) service_pool.cc:225] Handling call
1026 02:56:35.464233 (+4089871us) ts_tablet_manager.cc:1151] Deleting tablet 9a345fa5edbd48d2891b8b40b09cb3f8
1026 02:56:35.464245 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.464359 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4089602}
W20251026 02:56:35.465945 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 9a345fa5edbd48d2891b8b40b09cb3f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a345fa5edbd48d2891b8b40b09cb3f8 already in progress: creating tablet
I20251026 02:56:35.466485 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 477) took 4090 ms. Trace:
I20251026 02:56:35.445505 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 193e32898b164133b954409a08c298ce (DEFAULT_TABLE table=default.Table_f78068ab6b104363a2ed09be81de7ea2 [id=52c2da2abb1e4417bd60703a5f672e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.467079 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 193e32898b164133b954409a08c298ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.455200 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 3efafaac6c634d088396d56c8debd990 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:35.456969 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 8e3cdb1f81214337a7ccc7816755f58c (DEFAULT_TABLE table=default.Table_55f34abf69314dfab04ededf15472ee8 [id=497a284ac16f4f6e9fcad5fe49dcbe13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.469262 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3cdb1f81214337a7ccc7816755f58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.459841 2378 tablet_bootstrap.cc:492] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:35.460894 1947 tablet_service.cc:1552] Processing DeleteTablet for tablet bc4ca4423e3a4e0c96846babec931fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:35.473151 2083 consensus_meta.cc:287] T 6cbea840b88c4ef6b140a9a769fd69df P 0b6b0a6c4e144982b67980118cddba48: Time spent flushing consensus metadata: real 0.859s user 0.005s sys 0.000s
I20251026 02:56:35.473762 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 0dea6ffb68834373a03ced4f0f804473 (DEFAULT_TABLE table=default.Table_02d48c3527e64c1e9f77b7a9d89ab372 [id=b3f1207ea9df4819b2176948c552b0c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.474676 1951 tablet_service.cc:1505] Processing CreateTablet for tablet c95cf788ecfd4ed491ff49f9f2dc8db3 (DEFAULT_TABLE table=default.Table_55b76386af9a4934813a5c92774f5697 [id=89d5f588892a42498df49590ec6e2f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.466602 2376 rpcz_store.cc:276] 1026 02:56:31.375677 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.375811 (+ 134us) service_pool.cc:225] Handling call
1026 02:56:35.466356 (+4090545us) ts_tablet_manager.cc:1151] Deleting tablet a3af2695c41a4439bc4092b7cb1b2edf
1026 02:56:35.466366 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.466465 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4090277}
W20251026 02:56:35.475569 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet a3af2695c41a4439bc4092b7cb1b2edf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3af2695c41a4439bc4092b7cb1b2edf already in progress: creating tablet
I20251026 02:56:35.477088 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 478) took 4088 ms. Trace:
I20251026 02:56:35.477221 2376 rpcz_store.cc:276] 1026 02:56:31.388607 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.388686 (+ 79us) service_pool.cc:225] Handling call
1026 02:56:35.476826 (+4088140us) ts_tablet_manager.cc:1151] Deleting tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5
1026 02:56:35.476839 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.477065 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4087857}
W20251026 02:56:35.477802 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 already in progress: creating tablet
I20251026 02:56:35.478214 2378 ts_tablet_manager.cc:1403] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 1.099s user 0.005s sys 0.007s
I20251026 02:56:35.480624 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dea6ffb68834373a03ced4f0f804473. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.482084 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 0245af5a2dcb4f569b62307965895c1f (DEFAULT_TABLE table=default.Table_da48008b6d1a4e0b8dd4f80c2bfad18a [id=5973911f5601469c87eb1bd1cc10cf0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.491848 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0245af5a2dcb4f569b62307965895c1f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:35.492477 1648 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 193e32898b164133b954409a08c298ce (table default.Table_6193b9cb563841bcbf42d20cf7a21f59 [id=52c2da2abb1e4417bd60703a5f672e70], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 193e32898b164133b954409a08c298ce consensus state has no leader
I20251026 02:56:35.493768 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803248412240468ea45131299f2bd5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.489430 2083 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43402 (request call id 622) took 1071 ms. Trace:
I20251026 02:56:35.494745 2083 rpcz_store.cc:276] 1026 02:56:34.417785 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.417967 (+ 182us) service_pool.cc:225] Handling call
1026 02:56:34.418506 (+ 539us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:34.418534 (+ 28us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:34.613900 (+195366us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.473347 (+859447us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:35.489402 (+ 16055us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:35.495036 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 902cc4f9dbbd4d8baa829448f2a4687a
I20251026 02:56:35.495317 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 88b0cca095514614817346d6eb9014f5 (DEFAULT_TABLE table=default.Table_7a84416153d6424eb94a56932605f7ad [id=8e6c217882d441c7a218e38985b0a157]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:35.496061 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23062e7292f44ad5ad9dfc22a401741c consensus state has no leader
I20251026 02:56:35.503219 1949 tablet_service.cc:1552] Processing DeleteTablet for tablet 902cc4f9dbbd4d8baa829448f2a4687a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:35.503803 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95cf788ecfd4ed491ff49f9f2dc8db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.462328 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 1219e8bdfde9444f9bfdf1d626389e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:35.478691 1947 tablet_service.cc:1505] Processing CreateTablet for tablet eff2a8d01a004fc89ba3421e10da6ff5 (DEFAULT_TABLE table=default.Table_7b456a8685d84372a3b4710e3ad99da5 [id=6e0b679cbe704ba084b2cda4b09fc488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.505249 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 488) took 3725 ms. Trace:
I20251026 02:56:35.516727 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_684ec52e10144760980fb35e3d5bad26" } new_table_name: "default.Table_4c6dfbbb86f14ddb8ca1ae3b8a51f70b" modify_external_catalogs: true
I20251026 02:56:35.518484 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_07ffc3d487ac48ff849a6272f9f9877e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.519156 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07ffc3d487ac48ff849a6272f9f9877e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.519173 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 77ecdcbe25e24267bf1066a1baad804a (DEFAULT_TABLE table=default.Table_1abe312e689149299920c58ec6f1a35d [id=e652cb77eee74028b76a98c922b82f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.519573 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77ecdcbe25e24267bf1066a1baad804a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.505394 2376 rpcz_store.cc:276] 1026 02:56:31.779880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.780626 (+ 746us) service_pool.cc:225] Handling call
1026 02:56:35.505107 (+3724481us) ts_tablet_manager.cc:1151] Deleting tablet c31d2704d1f14ddead69c794cc631ec1
1026 02:56:35.505117 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.505226 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3724222}
I20251026 02:56:35.525192 2376 tablet_replica.cc:333] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:35.525486 2376 raft_consensus.cc:2243] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:35.525723 2376 raft_consensus.cc:2272] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:35.526374 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88b0cca095514614817346d6eb9014f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.532272 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5e91141bf6d441ab9d1842e8a859869a" } new_table_name: "default.Table_907991c2b0034c118065ead079f273f9" modify_external_catalogs: true
I20251026 02:56:35.533407 2376 ts_tablet_manager.cc:1916] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:35.535121 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_96ed23975f9842f9a636936ad4f2727d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.535866 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96ed23975f9842f9a636936ad4f2727d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.538887 1962 tablet_service.cc:1505] Processing CreateTablet for tablet ba08469339184b11b9ced9e6bdd63d42 (DEFAULT_TABLE table=default.Table_fbd4d396350c48129da0be37e6c3bfc8 [id=0fa207476cf042a6a1a7d37c668aa9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.539389 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba08469339184b11b9ced9e6bdd63d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.509943 3320 raft_consensus.cc:493] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:35.540153 3320 raft_consensus.cc:515] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:35.542080 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eff2a8d01a004fc89ba3421e10da6ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.543332 2093 tablet_service.cc:1505] Processing CreateTablet for tablet d85848daf97044e980060490a145400c (DEFAULT_TABLE table=default.Table_067a40b223504137ad333312ad69073a [id=c5e5c1368be1483d979623e3bff6a485]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.543933 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d85848daf97044e980060490a145400c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.545615 2372 ts_tablet_manager.cc:1929] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251026 02:56:35.548661 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 035724949fe142ea93cb998f8fdc4344 consensus state has no leader
I20251026 02:56:35.549239 2372 log.cc:1199] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/f85c2745a371455a8dda67f31a19c9d5
I20251026 02:56:35.555079 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 193e32898b164133b954409a08c298ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.554669 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e62064fbdffb4954a3a1227efd6327ee" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:35.558357 2257 raft_consensus.cc:3060] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:35.566551 2381 consensus_queue.cc:260] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.567245 2381 ts_tablet_manager.cc:1434] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.363s user 0.001s sys 0.001s
I20251026 02:56:35.572880 2100 tablet_service.cc:1505] Processing CreateTablet for tablet e32a1f8dcad4430eb35d388bfad7d110 (DEFAULT_TABLE table=default.Table_7921be016a944a41953b751ba927dacd [id=d3c0e21fa8bd430ea09e47bb699e3704]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.559208 2378 raft_consensus.cc:359] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.574034 2378 raft_consensus.cc:385] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:35.574107 2378 raft_consensus.cc:740] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:35.574386 2378 consensus_queue.cc:260] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.575510 2085 tablet_service.cc:1505] Processing CreateTablet for tablet bd11b620e78c499ebaa1b35f361242c3 (DEFAULT_TABLE table=default.Table_b51c545ae1d248b293e87daf463aaf8f [id=9158d553997949bfbe37cf153f6b1ab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.580480 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f197526e7d34f849dbb4f28d8d414cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.581374 2381 tablet_bootstrap.cc:492] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:35.582062 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8b019eeaa6074ffb87c0935572a36ed7" } new_table_name: "default.Table_e586d81375544236ac7d95297fca01a8" modify_external_catalogs: true
I20251026 02:56:35.583295 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_03f92e9829754bccb37129e4415098d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.583925 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03f92e9829754bccb37129e4415098d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.594739 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d13a1abe2fae4061b799c01104b52dd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.595515 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d13a1abe2fae4061b799c01104b52dd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:35.604319 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 69689.9us
I20251026 02:56:35.611639 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_37d7dc70b2c44c8ea6a882e8bbb8c5d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.612432 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37d7dc70b2c44c8ea6a882e8bbb8c5d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.616588 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1dbbfb77851d443f8cf4d798f3e07e59
I20251026 02:56:35.621493 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7dc4340a68b546eba896325bdae7e3db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.622236 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dc4340a68b546eba896325bdae7e3db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.622985 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 8aa0575af07e433c8d96acab31edb4af (DEFAULT_TABLE table=default.Table_901960ec4b4749f7b73a558f65408857 [id=232e25932feb4f6bbb665815b97899cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.623570 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 803248412240468ea45131299f2bd5df (DEFAULT_TABLE table=default.Table_84ccb1774a294a1b823ffb7ee68e621f [id=b22fc534ade646fca6efe6af4e588c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.624603 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dbbfb77851d443f8cf4d798f3e07e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:35.624181 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 8e3cdb1f81214337a7ccc7816755f58c (DEFAULT_TABLE table=default.Table_55f34abf69314dfab04ededf15472ee8 [id=497a284ac16f4f6e9fcad5fe49dcbe13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.624958 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3cdb1f81214337a7ccc7816755f58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.625263 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 51d09bf439244c5985d056c06559779c (DEFAULT_TABLE table=default.Table_6722958282e540edaae2edf477b15a85 [id=b81fbec0dc3247bd9b516e995c233e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.625710 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d09bf439244c5985d056c06559779c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.627351 2378 ts_tablet_manager.cc:1434] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.149s user 0.001s sys 0.001s
I20251026 02:56:35.627709 2378 tablet_bootstrap.cc:492] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:35.624594 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 193e32898b164133b954409a08c298ce (DEFAULT_TABLE table=default.Table_f78068ab6b104363a2ed09be81de7ea2 [id=52c2da2abb1e4417bd60703a5f672e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.628635 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 193e32898b164133b954409a08c298ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.632707 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd11b620e78c499ebaa1b35f361242c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.639791 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet bc4ca4423e3a4e0c96846babec931fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:35.640203 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet a74d6a4e041347578b0976a8b54bf22c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:35.640450 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet 657e07d0b17c44d4b78b97210f17ce40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:35.640723 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet 3efafaac6c634d088396d56c8debd990 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:35.640900 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet 1219e8bdfde9444f9bfdf1d626389e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:35.641815 2096 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43402 (request call id 644) took 1033 ms. Trace:
I20251026 02:56:35.649771 2096 rpcz_store.cc:276] 1026 02:56:34.608641 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.953464 (+344823us) service_pool.cc:225] Handling call
1026 02:56:34.988563 (+ 35099us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:34.988588 (+ 25us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.511288 (+522700us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.574621 (+ 63333us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:35.641798 (+ 67177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:35.642220 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32a1f8dcad4430eb35d388bfad7d110. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.646292 2090 tablet_service.cc:1505] Processing CreateTablet for tablet c95cf788ecfd4ed491ff49f9f2dc8db3 (DEFAULT_TABLE table=default.Table_55b76386af9a4934813a5c92774f5697 [id=89d5f588892a42498df49590ec6e2f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.651002 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa0575af07e433c8d96acab31edb4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.651546 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803248412240468ea45131299f2bd5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.648074 2372 ts_tablet_manager.cc:1950] T f85c2745a371455a8dda67f31a19c9d5 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:35.652012 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 516) took 4001 ms. Trace:
I20251026 02:56:35.621214 3320 leader_election.cc:290] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:35.659541 2097 tablet_service.cc:1505] Processing CreateTablet for tablet b8a1244c1c49471fb7711a2ca9b89a99 (DEFAULT_TABLE table=default.Table_dcdbf613e1704183b39ba6294a0b202d [id=48076cd17ca94dd8928115777232e240]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:35.660970 2113 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.011s sys 0.005s
I20251026 02:56:35.663098 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8a1244c1c49471fb7711a2ca9b89a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.663431 3320 raft_consensus.cc:493] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:35.663581 3320 raft_consensus.cc:515] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:35.664301 3320 leader_election.cc:290] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:35.665416 3320 raft_consensus.cc:493] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:35.665549 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1ffead7ca2d946c78aa4b8a602865fc8
I20251026 02:56:35.665545 3320 raft_consensus.cc:515] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.666033 3320 leader_election.cc:290] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:35.667104 2053 leader_election.cc:343] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:35.667207 2053 leader_election.cc:304] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:35.667416 3320 raft_consensus.cc:2749] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:35.667802 2053 leader_election.cc:343] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:35.667894 2053 leader_election.cc:304] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:35.668115 3320 raft_consensus.cc:2749] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:35.671788 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ffead7ca2d946c78aa4b8a602865fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:35.674549 1663 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 5226d4516ce24ecda51dab35c86da132 (table default.Table_b32c2e53d837476d9a0d5ae8ea1843d0 [id=c8435cef755d4de69c33f599df1c0f75]). Expected version 4 got 2
I20251026 02:56:35.674831 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 0245af5a2dcb4f569b62307965895c1f (DEFAULT_TABLE table=default.Table_da48008b6d1a4e0b8dd4f80c2bfad18a [id=5973911f5601469c87eb1bd1cc10cf0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.652158 2372 rpcz_store.cc:276] 1026 02:56:31.650475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.650580 (+ 105us) service_pool.cc:225] Handling call
1026 02:56:34.978716 (+3328136us) ts_tablet_manager.cc:1151] Deleting tablet f85c2745a371455a8dda67f31a19c9d5
1026 02:56:34.978727 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.490721 (+511994us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.545517 (+ 54796us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.651994 (+106477us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3327893}
I20251026 02:56:35.676527 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet f85c2745a371455a8dda67f31a19c9d5 (table default.Table_542a136f8cdd4d32b2e1e6fbf7bcc448 [id=f802cc99a8ec439a85a18e9b9a2826e7]) successfully deleted
I20251026 02:56:35.679241 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b12aefa4eb5b4380a6b9a31290cd24dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.680279 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b12aefa4eb5b4380a6b9a31290cd24dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:35.682032 2053 leader_election.cc:343] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:35.682179 2053 leader_election.cc:304] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:35.682562 2053 leader_election.cc:343] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:35.684033 1976 raft_consensus.cc:3060] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:35.691960 2378 tablet_bootstrap.cc:654] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:35.694797 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 24f16e40a30c41788a6f9813c235c5ab
I20251026 02:56:35.703677 2257 raft_consensus.cc:2468] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:35.709002 2381 tablet_bootstrap.cc:654] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:35.713735 3245 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.012s sys 0.012s
I20251026 02:56:35.714386 2377 ts_tablet_manager.cc:1929] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:35.721928 3187 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 02:56:35.723062 3187 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 02:56:35.723448 3187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 02:56:35.730206 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 351cd60b076a49949f03640c981845c4
W20251026 02:56:35.731207 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cfac07917ee34e6dba3fac5e081f6b43 consensus state has no leader
W20251026 02:56:35.737634 1647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.60:33729, user_credentials={real_user=slave}} blocked reactor thread for 66848.4us
I20251026 02:56:35.682657 2053 leader_election.cc:304] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:35.738374 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 55810.3us
W20251026 02:56:35.740022 2113 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.958s user 0.012s sys 0.005s
I20251026 02:56:35.682771 3320 raft_consensus.cc:2749] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:35.741659 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_082149a0d40742629c17967fd62a789e" } new_table_name: "default.Table_05ee3e62f67e424989346a2bd6f8389b" modify_external_catalogs: true
I20251026 02:56:35.741829 1982 raft_consensus.cc:3060] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:35.741957 1977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6108219254bd448c852096c0c967384b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:35.742331 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a3491b303f94fe2a2c838297f73d464" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:35.742511 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b7d85a2941545ceaa16b22e8328b4ff" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:35.742683 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d65b0d79e2e54af9810022b683c3288b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:35.742838 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85f9fcae83a04c6e907c7e9c8cd95dbe" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:35.743158 1973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6f9ae327aa4401baf66a737eaa93cf6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:35.743335 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_adc311a814324b2db4e8aee6caa91f09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.743924 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adc311a814324b2db4e8aee6caa91f09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.744118 1947 tablet_service.cc:1552] Processing DeleteTablet for tablet 0efdcc36ff5b47379e0a76512e05d379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:23 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:35.744201 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 24f16e40a30c41788a6f9813c235c5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:35.745570 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8248296e6b4a45d497271230d96ac519"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.746140 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8248296e6b4a45d497271230d96ac519 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.744119 2376 ts_tablet_manager.cc:1929] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:35.748247 2376 log.cc:1199] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/4011930a277d410f863a1558416c4b0f
I20251026 02:56:35.748805 2376 ts_tablet_manager.cc:1950] T 4011930a277d410f863a1558416c4b0f P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:35.748148 2378 tablet_bootstrap.cc:492] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:35.750723 2378 ts_tablet_manager.cc:1403] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.123s user 0.006s sys 0.002s
I20251026 02:56:35.740836 2113 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.150.3:37881 (request call id 378) took 1041 ms. Trace:
I20251026 02:56:35.751762 2378 raft_consensus.cc:359] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.751945 2378 raft_consensus.cc:385] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:35.751998 2378 raft_consensus.cc:740] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:35.752300 2378 consensus_queue.cc:260] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.752825 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 489) took 3883 ms. Trace:
I20251026 02:56:35.753008 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 115ccccf16b74efc9f24b607cf0c74d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:35.752962 2376 rpcz_store.cc:276] 1026 02:56:31.869789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.869893 (+ 104us) service_pool.cc:225] Handling call
1026 02:56:35.525128 (+3655235us) ts_tablet_manager.cc:1151] Deleting tablet 4011930a277d410f863a1558416c4b0f
1026 02:56:35.525141 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.709389 (+184248us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.744013 (+ 34624us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.752803 (+ 8790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3654847}
I20251026 02:56:35.753908 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 496) took 3581 ms. Trace:
I20251026 02:56:35.754312 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 351cd60b076a49949f03640c981845c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:35.756561 2053 leader_election.cc:343] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:35.756764 2053 leader_election.cc:343] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:35.756917 2053 leader_election.cc:343] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:35.757068 2053 leader_election.cc:343] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:35.757203 2053 leader_election.cc:343] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:35.756776 2376 rpcz_store.cc:276] 1026 02:56:32.172544 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.172612 (+ 68us) service_pool.cc:225] Handling call
1026 02:56:35.753671 (+3581059us) ts_tablet_manager.cc:1151] Deleting tablet 9fb42b54f558499090a1d7cf257dd84c
1026 02:56:35.753684 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.753856 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3576126}
W20251026 02:56:35.757341 2053 leader_election.cc:343] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:35.757743 2378 ts_tablet_manager.cc:1434] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251026 02:56:35.757942 3006 raft_consensus.cc:2749] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:35.757870 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 4011930a277d410f863a1558416c4b0f (table default.Table_8dcca91177f5452e816e2080bceadb93 [id=30cb7a766d854568892ad3a733d34f49]) successfully deleted
I20251026 02:56:35.758376 2378 tablet_bootstrap.cc:492] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:35.758009 2376 tablet_replica.cc:333] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:35.758965 2376 raft_consensus.cc:2243] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:35.759162 2376 raft_consensus.cc:2272] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:35.761536 2378 tablet_bootstrap.cc:654] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:35.768868 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e75a50dda2414622962b8832fdf3cb83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.769672 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e75a50dda2414622962b8832fdf3cb83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.771796 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ad4b4db01c7d4fe0a4e99f2f54cee0e8" } modify_external_catalogs: true
I20251026 02:56:35.772069 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ad4b4db01c7d4fe0a4e99f2f54cee0e8" } modify_external_catalogs: true
I20251026 02:56:35.764315 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b0933f71cb4ac7a58986d16b613e64" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:35.773144 3250 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:35.759732 2376 ts_tablet_manager.cc:1916] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:35.775202 2053 leader_election.cc:343] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:35.777199 2095 tablet_service.cc:1505] Processing CreateTablet for tablet eff2a8d01a004fc89ba3421e10da6ff5 (DEFAULT_TABLE table=default.Table_7b456a8685d84372a3b4710e3ad99da5 [id=6e0b679cbe704ba084b2cda4b09fc488]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:35.782634 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f719c720cc474181a63786a3e248ab2c consensus state has no leader
I20251026 02:56:35.783958 2378 tablet_bootstrap.cc:492] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:35.784124 2378 ts_tablet_manager.cc:1403] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20251026 02:56:35.784922 2378 raft_consensus.cc:359] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.785096 2378 raft_consensus.cc:385] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:35.785153 2378 raft_consensus.cc:740] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:35.785365 2378 consensus_queue.cc:260] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.789489 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_51c1281363e94fcaa024e7e1aca9b333" } new_table_name: "default.Table_2061f39f1e524edb8d9824942b2ef13d" modify_external_catalogs: true
I20251026 02:56:35.795763 2376 ts_tablet_manager.cc:1929] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:35.803722 2376 log.cc:1199] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/9156314f98ac4ed9981f6f13fc585043
I20251026 02:56:35.804889 2376 ts_tablet_manager.cc:1950] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:35.796653 2378 ts_tablet_manager.cc:1434] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251026 02:56:35.751760 2113 rpcz_store.cc:276] 1026 02:56:34.699220 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.702980 (+ 3760us) service_pool.cc:225] Handling call
1026 02:56:34.703065 (+ 85us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 02:56:34.703179 (+ 114us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 02:56:34.703183 (+ 4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 02:56:34.703344 (+ 161us) log.cc:844] Serialized 26 byte log entry
1026 02:56:35.740523 (+1037179us) raft_consensus.cc:1645] Marking committed up to 0
1026 02:56:35.740541 (+ 18us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 02:56:35.740638 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 02:56:35.740651 (+ 13us) raft_consensus.cc:1685] finished
1026 02:56:35.740655 (+ 4us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 02:56:35.740808 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5029,"raft.run_cpu_time_us":22,"raft.run_wall_time_us":22,"thread_start_us":1036739,"threads_started":1,"wal-append.queue_time_us":734705}
I20251026 02:56:35.805891 2378 tablet_bootstrap.cc:492] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:35.807868 2378 tablet_bootstrap.cc:654] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:35.819517 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 492) took 3666 ms. Trace:
I20251026 02:56:35.820811 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0245af5a2dcb4f569b62307965895c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.821035 2376 rpcz_store.cc:276] 1026 02:56:32.152729 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.153408 (+ 679us) service_pool.cc:225] Handling call
1026 02:56:35.757957 (+3604549us) ts_tablet_manager.cc:1151] Deleting tablet 9156314f98ac4ed9981f6f13fc585043
1026 02:56:35.757969 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.779404 (+ 21435us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.795616 (+ 16212us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.819489 (+ 23873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3579853}
I20251026 02:56:35.821594 2376 tablet_replica.cc:333] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:35.821798 2372 tablet_replica.cc:333] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:35.822024 2372 raft_consensus.cc:2243] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:35.822253 1657 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 5226d4516ce24ecda51dab35c86da132 (table default.Table_b32c2e53d837476d9a0d5ae8ea1843d0 [id=c8435cef755d4de69c33f599df1c0f75]). Expected version 4 got 2
I20251026 02:56:35.822469 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dbbfb77851d443f8cf4d798f3e07e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:35.822687 2376 raft_consensus.cc:2243] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:35.822901 1657 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe (table default.Table_47994972d3e0469c9acc96120aad4e1f [id=87349d9c6c764382b828e716e7854c59]). Expected version 3 got 0
I20251026 02:56:35.823141 2376 raft_consensus.cc:2272] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:35.823233 1657 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 352a284902c94f088e2e6ae006162db0 (table default.Table_6cd4ee82e9ec406f971669b7f0722677 [id=414375003cd44b5c8c9f0249b339c28a]). Expected version 1 got 0
I20251026 02:56:35.823257 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ffead7ca2d946c78aa4b8a602865fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:35.823458 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 24f16e40a30c41788a6f9813c235c5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:35.823665 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 351cd60b076a49949f03640c981845c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:35.823979 2376 ts_tablet_manager.cc:1916] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:35.824146 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eff2a8d01a004fc89ba3421e10da6ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.821946 2378 tablet_bootstrap.cc:492] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:35.825245 2378 ts_tablet_manager.cc:1403] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20251026 02:56:35.826122 2378 raft_consensus.cc:359] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.826300 2378 raft_consensus.cc:385] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:35.826371 2378 raft_consensus.cc:740] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:35.826578 2378 consensus_queue.cc:260] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.828857 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_07ffc3d487ac48ff849a6272f9f9877e" } new_table_name: "default.Table_000637f9a96344c28b90e88c10c89a07" modify_external_catalogs: true
I20251026 02:56:35.828836 2099 tablet_service.cc:1505] Processing CreateTablet for tablet ba08469339184b11b9ced9e6bdd63d42 (DEFAULT_TABLE table=default.Table_fbd4d396350c48129da0be37e6c3bfc8 [id=0fa207476cf042a6a1a7d37c668aa9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.829365 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba08469339184b11b9ced9e6bdd63d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.830296 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95cf788ecfd4ed491ff49f9f2dc8db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.830385 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 0dea6ffb68834373a03ced4f0f804473 (DEFAULT_TABLE table=default.Table_02d48c3527e64c1e9f77b7a9d89ab372 [id=b3f1207ea9df4819b2176948c552b0c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.830685 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dea6ffb68834373a03ced4f0f804473. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.831928 2372 raft_consensus.cc:2272] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:35.832758 2372 ts_tablet_manager.cc:1916] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:35.833073 2378 ts_tablet_manager.cc:1434] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251026 02:56:35.833420 2378 tablet_bootstrap.cc:492] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:35.834584 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 51d09bf439244c5985d056c06559779c (DEFAULT_TABLE table=default.Table_6722958282e540edaae2edf477b15a85 [id=b81fbec0dc3247bd9b516e995c233e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.835033 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d09bf439244c5985d056c06559779c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.837317 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet a6f9ae327aa4401baf66a737eaa93cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:35.842183 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 9fb42b54f558499090a1d7cf257dd84c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fb42b54f558499090a1d7cf257dd84c already in progress: creating tablet
I20251026 02:56:35.843214 3347 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:35.843653 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 9156314f98ac4ed9981f6f13fc585043 (table default.Table_4578a4ce81ea42cd878859a8167babff [id=f914c9974ac44d3f84c3d312649fe255]) successfully deleted
I20251026 02:56:35.848160 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ff7d398f97d04b29873f9eacd1c9fb11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:35.850085 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff7d398f97d04b29873f9eacd1c9fb11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:35.848873 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_95868643edf84f8a8a325f3cb56aa732" } modify_external_catalogs: true
I20251026 02:56:35.852124 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_95868643edf84f8a8a325f3cb56aa732" } modify_external_catalogs: true
I20251026 02:56:35.852655 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 902cc4f9dbbd4d8baa829448f2a4687a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:35.862138 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a3491b303f94fe2a2c838297f73d464 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b5e61afe90e4bc4a2897de21e0b3919) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:35.877595 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ceb7249968ff44c1b5be1d1d18605ffe consensus state has no leader
I20251026 02:56:35.878157 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_60d710dc906a41308e183f47acdda437" } modify_external_catalogs: true
I20251026 02:56:35.878441 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_60d710dc906a41308e183f47acdda437" } modify_external_catalogs: true
W20251026 02:56:35.841491 1974 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.189s user 0.028s sys 0.106s
W20251026 02:56:35.883517 1974 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.231s user 0.028s sys 0.106s
I20251026 02:56:35.885897 1974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.150.3:55185 (request call id 328) took 1281 ms. Trace:
I20251026 02:56:35.889606 1974 rpcz_store.cc:276] 1026 02:56:34.651113 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.651354 (+ 241us) service_pool.cc:225] Handling call
1026 02:56:34.651545 (+ 191us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 02:56:34.651686 (+ 141us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 02:56:34.651690 (+ 4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 02:56:34.651824 (+ 134us) alter_schema_op.cc:128] START. Timestamp: 1761447394127737
1026 02:56:34.652081 (+ 257us) log.cc:844] Serialized 173 byte log entry
1026 02:56:35.885774 (+1233693us) raft_consensus.cc:1645] Marking committed up to 3
1026 02:56:35.885783 (+ 9us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 02:56:35.885808 (+ 25us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 02:56:35.885815 (+ 7us) raft_consensus.cc:1685] finished
1026 02:56:35.885818 (+ 3us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 02:56:35.885873 (+ 55us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:34.665252 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1026 02:56:34.665383 (+ 131us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1026 02:56:34.665395 (+ 12us) alter_schema_op.cc:66] Acquired schema lock
1026 02:56:34.665404 (+ 9us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
Metrics: {"thread_start_us":1233618,"threads_started":1,"wal-append.queue_time_us":798385,"child_traces":[["op",{"prepare.queue_time_us":13318,"prepare.run_cpu_time_us":181,"prepare.run_wall_time_us":183}]]}
I20251026 02:56:35.907088 1938 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58886 (request call id 123) took 2239 ms. Trace:
I20251026 02:56:35.925688 1938 rpcz_store.cc:276] 1026 02:56:33.667739 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.669338 (+ 1599us) service_pool.cc:225] Handling call
1026 02:56:34.725601 (+1056263us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x6160000d2d54. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.112579 (+386978us) spinlock_profiling.cc:243] Waited 8.36 ms on lock 0x6160000b6630. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc4273ac829 00007fc429ed0478 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.892378 (+779799us) spinlock_profiling.cc:243] Waited 773 ms on lock 0x6160000b3854. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.907049 (+ 14671us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4202601472}
I20251026 02:56:35.912822 2378 tablet_bootstrap.cc:654] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:35.918586 1942 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58886 (request call id 125) took 2221 ms. Trace:
I20251026 02:56:35.948447 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_dcdbf613e1704183b39ba6294a0b202d" } modify_external_catalogs: true
I20251026 02:56:35.948719 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_dcdbf613e1704183b39ba6294a0b202d" } modify_external_catalogs: true
I20251026 02:56:35.947642 1942 rpcz_store.cc:276] 1026 02:56:33.696915 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.700088 (+ 3173us) service_pool.cc:225] Handling call
1026 02:56:34.732600 (+1032512us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x6160000d2d54. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.104362 (+371762us) spinlock_profiling.cc:243] Waited 7.87 ms on lock 0x6160000b6630. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc4273ac829 00007fc429ed0478 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.113632 (+ 9270us) spinlock_profiling.cc:243] Waited 9.08 ms on lock 0x6160000b6630. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc4273ad8ec 00007fc4273ace1a 00007fc429ed0478 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8
1026 02:56:35.894275 (+780643us) spinlock_profiling.cc:243] Waited 772 ms on lock 0x6160000b3854. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.918539 (+ 24264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4180895872}
I20251026 02:56:35.888231 2376 ts_tablet_manager.cc:1929] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:35.950210 2376 log.cc:1199] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/f8533f1c191042ee9f2c0b99972fe931
I20251026 02:56:35.950567 2376 ts_tablet_manager.cc:1950] T f8533f1c191042ee9f2c0b99972fe931 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:35.950896 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 491) took 3798 ms. Trace:
I20251026 02:56:35.950968 2376 rpcz_store.cc:276] 1026 02:56:32.152629 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.153377 (+ 748us) service_pool.cc:225] Handling call
1026 02:56:35.821547 (+3668170us) ts_tablet_manager.cc:1151] Deleting tablet f8533f1c191042ee9f2c0b99972fe931
1026 02:56:35.821557 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.837603 (+ 16046us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.888097 (+ 50494us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.950876 (+ 62779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3641132}
I20251026 02:56:35.951458 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 497) took 3778 ms. Trace:
I20251026 02:56:35.951471 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 30c619ad37364747b6ebf97a73ef45a2
I20251026 02:56:35.951541 2376 rpcz_store.cc:276] 1026 02:56:32.172687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.173528 (+ 841us) service_pool.cc:225] Handling call
1026 02:56:35.951285 (+3777757us) ts_tablet_manager.cc:1151] Deleting tablet 21b936fa4f6a4ff6af321971a30cccbe
1026 02:56:35.951298 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.951439 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3770102}
I20251026 02:56:35.951898 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 499) took 3769 ms. Trace:
I20251026 02:56:35.951967 2376 rpcz_store.cc:276] 1026 02:56:32.182430 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.182501 (+ 71us) service_pool.cc:225] Handling call
1026 02:56:35.951789 (+3769288us) ts_tablet_manager.cc:1151] Deleting tablet 4a3f3fc3c7a04d05bae28472251f3acc
1026 02:56:35.951798 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.951884 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3769060}
I20251026 02:56:35.952337 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 498) took 3770 ms. Trace:
I20251026 02:56:35.952415 2376 rpcz_store.cc:276] 1026 02:56:32.182284 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.188287 (+ 6003us) service_pool.cc:225] Handling call
1026 02:56:35.952197 (+3763910us) ts_tablet_manager.cc:1151] Deleting tablet 027dd92fc5cf4c9e847de3e55e0dac1c
1026 02:56:35.952231 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.952321 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3758828}
I20251026 02:56:35.952730 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 500) took 3765 ms. Trace:
I20251026 02:56:35.952800 2376 rpcz_store.cc:276] 1026 02:56:32.187638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.187936 (+ 298us) service_pool.cc:225] Handling call
1026 02:56:35.952635 (+3764699us) ts_tablet_manager.cc:1151] Deleting tablet 63a63b796191465aab9a63d3ffb0554d
1026 02:56:35.952644 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.952718 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3759163}
I20251026 02:56:35.953055 2376 tablet_replica.cc:333] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:35.953292 2376 raft_consensus.cc:2243] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:35.953511 2376 raft_consensus.cc:2272] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:35.954183 2376 ts_tablet_manager.cc:1916] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:35.924088 1941 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58886 (request call id 124) took 2228 ms. Trace:
I20251026 02:56:35.957052 1941 rpcz_store.cc:276] 1026 02:56:33.695674 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.696259 (+ 585us) service_pool.cc:225] Handling call
1026 02:56:34.729393 (+1033134us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x6160000d2d54. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.888963 (+1159570us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x6160000b3854. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.924036 (+ 35073us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4312587264}
I20251026 02:56:35.924853 1940 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58886 (request call id 122) took 2346 ms. Trace:
I20251026 02:56:35.958639 1940 rpcz_store.cc:276] 1026 02:56:33.578160 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.578242 (+ 82us) service_pool.cc:225] Handling call
1026 02:56:34.728987 (+1150745us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x6160000d2d54. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.889984 (+1160997us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x6160000b3854. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.924808 (+ 34824us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4637441152}
I20251026 02:56:35.958853 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet f8533f1c191042ee9f2c0b99972fe931 (table default.Table_e70cf6d4e0dc4017b484753f4b0250d9 [id=f4fb24ed75a24d81ae8f90619ded2ca9]) successfully deleted
W20251026 02:56:35.959218 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 21b936fa4f6a4ff6af321971a30cccbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21b936fa4f6a4ff6af321971a30cccbe already in progress: creating tablet
W20251026 02:56:35.959455 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 4a3f3fc3c7a04d05bae28472251f3acc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a3f3fc3c7a04d05bae28472251f3acc already in progress: creating tablet
W20251026 02:56:35.959694 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 027dd92fc5cf4c9e847de3e55e0dac1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 027dd92fc5cf4c9e847de3e55e0dac1c already in progress: creating tablet
W20251026 02:56:35.959888 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 63a63b796191465aab9a63d3ffb0554d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63a63b796191465aab9a63d3ffb0554d already in progress: creating tablet
I20251026 02:56:35.912091 1939 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58886 (request call id 126) took 2094 ms. Trace:
I20251026 02:56:35.960147 1939 rpcz_store.cc:276] 1026 02:56:33.818059 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.818123 (+ 64us) service_pool.cc:225] Handling call
1026 02:56:34.287726 (+469603us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x616000125254. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:34.727072 (+439346us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x6160000d2d54. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.088679 (+361607us) spinlock_profiling.cc:243] Waited 9.74 ms on lock 0x6160000b6630. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc4273ac829 00007fc429ed0478 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.113139 (+ 24460us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x6160000b6630. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc4273ad8ec 00007fc4273ace1a 00007fc429ed0478 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8
1026 02:56:35.896042 (+782903us) spinlock_profiling.cc:243] Waited 772 ms on lock 0x6160000b3854. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:35.912056 (+ 16014us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3839166336}
I20251026 02:56:35.960366 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2b6ab2f608284fb1aec83b83ab7da74f" } new_table_name: "default.Table_5710a38a4bd74f5a8a6818bbd7bcae5f" modify_external_catalogs: true
I20251026 02:56:35.962826 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b56f9f35731e4bc4be24623e8e79fe5e" } modify_external_catalogs: true
I20251026 02:56:35.963043 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b56f9f35731e4bc4be24623e8e79fe5e" } modify_external_catalogs: true
I20251026 02:56:35.964032 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 30c619ad37364747b6ebf97a73ef45a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:35.970490 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64415f5588db4df0a94676b650fb3631 consensus state has no leader
I20251026 02:56:35.973402 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 64415f5588db4df0a94676b650fb3631 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:35.713294 2194 thread.cc:641] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.001s sys 0.012s
W20251026 02:56:35.977413 2194 thread.cc:608] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.801s user 0.001s sys 0.013s
I20251026 02:56:35.978511 3028 raft_consensus.cc:493] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:35.714093 3245 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s user 0.012s sys 0.012s
I20251026 02:56:35.978638 3028 raft_consensus.cc:515] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:35.979425 3028 leader_election.cc:290] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:35.714943 2228 tablet_service.cc:1505] Processing CreateTablet for tablet c95cf788ecfd4ed491ff49f9f2dc8db3 (DEFAULT_TABLE table=default.Table_55b76386af9a4934813a5c92774f5697 [id=89d5f588892a42498df49590ec6e2f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.979914 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95cf788ecfd4ed491ff49f9f2dc8db3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:35.731503 2239 consensus_meta.cc:287] T 64faa109f006414bb955ed483ecc189e P 696476265495492790739ec126a08539: Time spent flushing consensus metadata: real 0.553s user 0.005s sys 0.000s
W20251026 02:56:35.731851 2225 consensus_meta.cc:287] T 88b0cca095514614817346d6eb9014f5 P 696476265495492790739ec126a08539: Time spent flushing consensus metadata: real 0.520s user 0.003s sys 0.000s
W20251026 02:56:35.740482 3144 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s user 0.007s sys 0.016s
W20251026 02:56:35.982002 3144 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.981s user 0.007s sys 0.016s
I20251026 02:56:35.983117 1978 raft_consensus.cc:3060] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:35.985244 2239 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35450 (request call id 660) took 1182 ms. Trace:
I20251026 02:56:35.985319 2239 rpcz_store.cc:276] 1026 02:56:34.802386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.089623 (+287237us) service_pool.cc:225] Handling call
1026 02:56:35.174812 (+ 85189us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.174845 (+ 33us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.178317 (+ 3472us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.985081 (+806764us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:35.985225 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:35.985719 2225 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35450 (request call id 662) took 1183 ms. Trace:
I20251026 02:56:35.985791 2225 rpcz_store.cc:276] 1026 02:56:34.802553 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.111490 (+308937us) service_pool.cc:225] Handling call
1026 02:56:35.180730 (+ 69240us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.180749 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.183955 (+ 3206us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.985568 (+801613us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:35.985683 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3538}
I20251026 02:56:35.885185 2381 tablet_bootstrap.cc:492] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:36.002547 2381 ts_tablet_manager.cc:1403] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.421s user 0.007s sys 0.007s
I20251026 02:56:36.003602 2381 raft_consensus.cc:359] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:36.004148 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6dbe3f1f3e074da5bae9badcffa01f6b has no consensus state
I20251026 02:56:36.005676 2381 raft_consensus.cc:385] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.005777 2381 raft_consensus.cc:740] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.006009 2381 consensus_queue.cc:260] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.006594 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 865fbd674db34f668fb5a2f5836a9b43
I20251026 02:56:36.006664 2381 ts_tablet_manager.cc:1434] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:36.006989 2381 tablet_bootstrap.cc:492] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:36.008819 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 865fbd674db34f668fb5a2f5836a9b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:36.016067 1629 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 76971.4us
I20251026 02:56:36.017269 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_39b5d99fe09d427389ea14393b4f56ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:36.018110 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39b5d99fe09d427389ea14393b4f56ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.022661 3144 raft_consensus.cc:493] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.022749 2100 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43402 (request call id 671) took 1063 ms. Trace:
I20251026 02:56:36.022864 3144 raft_consensus.cc:515] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.022874 2100 rpcz_store.cc:276] 1026 02:56:34.958770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.542560 (+583790us) service_pool.cc:225] Handling call
1026 02:56:35.641891 (+ 99331us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.641948 (+ 57us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.827994 (+186046us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.879370 (+ 51376us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:36.022722 (+143352us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:36.023610 3144 leader_election.cc:290] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:36.023972 2092 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43402 (request call id 675) took 1064 ms. Trace:
I20251026 02:56:36.024163 2092 rpcz_store.cc:276] 1026 02:56:34.959187 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.548251 (+589064us) service_pool.cc:225] Handling call
1026 02:56:35.644800 (+ 96549us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.644839 (+ 39us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.817733 (+172894us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.873129 (+ 55396us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:36.023951 (+150822us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:36.028664 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3
W20251026 02:56:36.028645 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 61182.9us
W20251026 02:56:36.028990 2195 consensus_peers.cc:597] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f85c2745a371455a8dda67f31a19c9d5. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:36.031241 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 30c619ad37364747b6ebf97a73ef45a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:36.040484 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b8a1244c1c49471fb7711a2ca9b89a99
I20251026 02:56:36.047385 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9f48bbcff2cc4a47b64cb3862caccc33" } new_table_name: "default.Table_2ec6f8b4f5a1475c91edabb9dbceced9" modify_external_catalogs: true
I20251026 02:56:36.049909 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 865fbd674db34f668fb5a2f5836a9b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.050161 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7a07c4cc0b48429aad305ac537734fd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:35.893501 2231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35450 (request call id 659) took 1091 ms. Trace:
I20251026 02:56:35.896555 2243 tablet_service.cc:1505] Processing CreateTablet for tablet eff2a8d01a004fc89ba3421e10da6ff5 (DEFAULT_TABLE table=default.Table_7b456a8685d84372a3b4710e3ad99da5 [id=6e0b679cbe704ba084b2cda4b09fc488]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:36.051003 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a07c4cc0b48429aad305ac537734fd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.051173 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eff2a8d01a004fc89ba3421e10da6ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.977427 2377 log.cc:1199] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/a70d70abc3a34dac8cf9a747903fa78c
I20251026 02:56:36.053762 2377 ts_tablet_manager.cc:1950] T a70d70abc3a34dac8cf9a747903fa78c P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:36.054057 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 474) took 5790 ms. Trace:
I20251026 02:56:36.054126 2377 rpcz_store.cc:276] 1026 02:56:30.263721 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.263771 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:35.187606 (+4923835us) ts_tablet_manager.cc:1151] Deleting tablet a70d70abc3a34dac8cf9a747903fa78c
1026 02:56:35.187613 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:35.617596 (+429983us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.714277 (+ 96681us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.054040 (+339763us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4923349}
I20251026 02:56:36.054567 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 490) took 5244 ms. Trace:
I20251026 02:56:36.054630 2377 rpcz_store.cc:276] 1026 02:56:30.809848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.809922 (+ 74us) service_pool.cc:225] Handling call
1026 02:56:36.054425 (+5244503us) ts_tablet_manager.cc:1151] Deleting tablet 1eabfc95caa84afc9a8956a7ac13fea6
1026 02:56:36.054433 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.054549 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5244206}
I20251026 02:56:36.054916 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 492) took 5152 ms. Trace:
I20251026 02:56:36.054979 2377 rpcz_store.cc:276] 1026 02:56:30.904649 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.904729 (+ 80us) service_pool.cc:225] Handling call
1026 02:56:36.054810 (+5150081us) ts_tablet_manager.cc:1151] Deleting tablet 4b49c0996c4d41cbaddcdf0b75846bcd
1026 02:56:36.054819 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.054901 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5149808}
I20251026 02:56:36.055238 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 493) took 5149 ms. Trace:
I20251026 02:56:36.055291 2377 rpcz_store.cc:276] 1026 02:56:30.906293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.906364 (+ 71us) service_pool.cc:225] Handling call
1026 02:56:36.055148 (+5148784us) ts_tablet_manager.cc:1151] Deleting tablet 7177931353164b2aa82c00b2807ccd63
1026 02:56:36.055156 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.055224 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5148490}
I20251026 02:56:36.055573 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 494) took 5089 ms. Trace:
I20251026 02:56:36.055626 2377 rpcz_store.cc:276] 1026 02:56:30.966204 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.966823 (+ 619us) service_pool.cc:225] Handling call
1026 02:56:36.055494 (+5088671us) ts_tablet_manager.cc:1151] Deleting tablet 8fe03778a1e842829cab25c90597a3cd
1026 02:56:36.055500 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.055562 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5087125}
I20251026 02:56:36.055830 2377 tablet_replica.cc:333] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:36.055984 2377 raft_consensus.cc:2243] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:36.056150 2377 raft_consensus.cc:2272] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:35.897081 2226 tablet_service.cc:1505] Processing CreateTablet for tablet 0245af5a2dcb4f569b62307965895c1f (DEFAULT_TABLE table=default.Table_da48008b6d1a4e0b8dd4f80c2bfad18a [id=5973911f5601469c87eb1bd1cc10cf0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.056559 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0245af5a2dcb4f569b62307965895c1f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:35.898792 2240 consensus_meta.cc:287] T bd11b620e78c499ebaa1b35f361242c3 P 696476265495492790739ec126a08539: Time spent flushing consensus metadata: real 0.699s user 0.002s sys 0.003s
I20251026 02:56:36.060048 2240 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35450 (request call id 666) took 1257 ms. Trace:
I20251026 02:56:36.060155 2240 rpcz_store.cc:276] 1026 02:56:34.802874 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.126313 (+323439us) service_pool.cc:225] Handling call
1026 02:56:35.196431 (+ 70118us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.196455 (+ 24us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.199828 (+ 3373us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.059837 (+860009us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:36.060027 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:36.060655 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:36.060746 1947 tablet_service.cc:1552] Processing DeleteTablet for tablet b8a1244c1c49471fb7711a2ca9b89a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:35.902845 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 0dea6ffb68834373a03ced4f0f804473 (DEFAULT_TABLE table=default.Table_02d48c3527e64c1e9f77b7a9d89ab372 [id=b3f1207ea9df4819b2176948c552b0c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.061012 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_720e8839bcb647ce8e030119c884cffa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:36.061431 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dea6ffb68834373a03ced4f0f804473. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.063580 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a70ab000815d4d91b69ae8a6727c6b88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:36.065228 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6f081e5839af41399767123dd4a4e5d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:35.915755 2233 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35450 (request call id 668) took 1112 ms. Trace:
I20251026 02:56:35.921103 2238 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35450 (request call id 670) took 1004 ms. Trace:
I20251026 02:56:36.065580 2238 rpcz_store.cc:276] 1026 02:56:34.916361 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.147833 (+231472us) service_pool.cc:225] Handling call
1026 02:56:35.217729 (+ 69896us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.217756 (+ 27us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.899505 (+681749us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.920909 (+ 21404us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:35.921070 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:36.065514 2233 rpcz_store.cc:276] 1026 02:56:34.803015 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.143417 (+340402us) service_pool.cc:225] Handling call
1026 02:56:35.204763 (+ 61346us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.204790 (+ 27us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.912928 (+708138us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.915581 (+ 2653us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:35.915733 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:35.924925 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35450 (request call id 661) took 1122 ms. Trace:
I20251026 02:56:35.933761 2237 tablet_service.cc:1505] Processing CreateTablet for tablet ba08469339184b11b9ced9e6bdd63d42 (DEFAULT_TABLE table=default.Table_fbd4d396350c48129da0be37e6c3bfc8 [id=0fa207476cf042a6a1a7d37c668aa9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.934109 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 902cc4f9dbbd4d8baa829448f2a4687a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:35.934577 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dbbfb77851d443f8cf4d798f3e07e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:35.934903 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 24f16e40a30c41788a6f9813c235c5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:35.935158 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 351cd60b076a49949f03640c981845c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:35.935165 2227 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ffead7ca2d946c78aa4b8a602865fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:35.935365 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 51d09bf439244c5985d056c06559779c (DEFAULT_TABLE table=default.Table_6722958282e540edaae2edf477b15a85 [id=b81fbec0dc3247bd9b516e995c233e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:35.963189 2258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b7d85a2941545ceaa16b22e8328b4ff" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:35.963831 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6108219254bd448c852096c0c967384b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:35.964496 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d65b0d79e2e54af9810022b683c3288b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:35.966572 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85f9fcae83a04c6e907c7e9c8cd95dbe" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:36.067371 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d09bf439244c5985d056c06559779c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:35.967021 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a3491b303f94fe2a2c838297f73d464" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:35.967329 2254 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b0933f71cb4ac7a58986d16b613e64" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:36.066623 2224 rpcz_store.cc:276] 1026 02:56:34.802468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.101923 (+299455us) service_pool.cc:225] Handling call
1026 02:56:35.179596 (+ 77673us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.179616 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.916984 (+737368us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.924775 (+ 7791us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:35.924906 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:36.069080 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba08469339184b11b9ced9e6bdd63d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.074385 3250 tablet.cc:1722] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:36.084131 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2e3bf8087ce7480584b9b66c8270c451" } new_table_name: "default.Table_81d15a2f4d424910a51421472c159f0d" modify_external_catalogs: true
W20251026 02:56:36.090349 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8306f2ece961480d906df88f40b9740b consensus state has no leader
I20251026 02:56:36.091148 2378 tablet_bootstrap.cc:492] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:36.091341 2378 ts_tablet_manager.cc:1403] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.258s user 0.013s sys 0.000s
I20251026 02:56:35.967559 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6f9ae327aa4401baf66a737eaa93cf6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:36.103150 2231 rpcz_store.cc:276] 1026 02:56:34.802288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.041049 (+238761us) service_pool.cc:225] Handling call
1026 02:56:35.310603 (+269554us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.310645 (+ 42us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.734988 (+424343us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:35.893304 (+158316us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:35.893477 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:36.106426 3144 raft_consensus.cc:493] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.106642 3144 raft_consensus.cc:515] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.107406 3144 leader_election.cc:290] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:36.117411 2377 ts_tablet_manager.cc:1916] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:36.120708 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a70ab000815d4d91b69ae8a6727c6b88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:36.122617 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_720e8839bcb647ce8e030119c884cffa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.122975 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e9b0e1d3f3f4fdbb8c58f26a4308771" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:36.123719 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 204836us
W20251026 02:56:36.123872 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f081e5839af41399767123dd4a4e5d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.124249 3320 raft_consensus.cc:493] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.124606 2091 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43402 (request call id 670) took 1165 ms. Trace:
I20251026 02:56:36.124684 2091 rpcz_store.cc:276] 1026 02:56:34.958701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.498488 (+539787us) service_pool.cc:225] Handling call
1026 02:56:35.519378 (+ 20890us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.519421 (+ 43us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.815899 (+296478us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.049658 (+233759us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:36.124592 (+ 74934us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:36.124985 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet b8a1244c1c49471fb7711a2ca9b89a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.125336 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7ffdaeec2240dcbe4cafdb96073a7d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:36.125563 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.124378 3320 raft_consensus.cc:515] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.129186 2378 raft_consensus.cc:359] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.129462 2378 raft_consensus.cc:385] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.129578 2378 raft_consensus.cc:740] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.129920 2378 consensus_queue.cc:260] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.130165 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 30c619ad37364747b6ebf97a73ef45a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.136328 3320 leader_election.cc:290] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:36.139341 1915 leader_election.cc:304] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:36.139815 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7cb8287939ea4ce3817a6043c996784e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:36.141059 3318 raft_consensus.cc:2804] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 02:56:36.141255 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cb8287939ea4ce3817a6043c996784e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.141898 3318 raft_consensus.cc:697] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:36.142172 3318 consensus_queue.cc:237] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:36.143549 1915 leader_election.cc:343] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 9e5bb0a6a9f947cf9ce119c1fc91fbf5
I20251026 02:56:36.146301 1915 leader_election.cc:304] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
W20251026 02:56:36.146486 1915 raft_consensus.cc:2686] T 9e5bb0a6a9f947cf9ce119c1fc91fbf5 P 7fef01f4f2b843538ea245a09dd087a8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251026 02:56:36.146548 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e9b0e1d3f3f4fdbb8c58f26a4308771" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:36.146729 2109 raft_consensus.cc:2468] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:36.147150 1915 leader_election.cc:343] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 9156314f98ac4ed9981f6f13fc585043
I20251026 02:56:36.147251 1915 leader_election.cc:304] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
W20251026 02:56:36.147418 1915 raft_consensus.cc:2686] T 9156314f98ac4ed9981f6f13fc585043 P 7fef01f4f2b843538ea245a09dd087a8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251026 02:56:36.148025 3006 raft_consensus.cc:493] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.148140 3006 raft_consensus.cc:515] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.148710 3006 leader_election.cc:290] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:36.150238 2378 ts_tablet_manager.cc:1434] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.059s user 0.002s sys 0.000s
I20251026 02:56:36.150923 2085 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43402 (request call id 673) took 1191 ms. Trace:
I20251026 02:56:36.150998 2085 rpcz_store.cc:276] 1026 02:56:34.958975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.547154 (+588179us) service_pool.cc:225] Handling call
1026 02:56:35.632456 (+ 85302us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.632502 (+ 46us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.883893 (+251391us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.150762 (+266869us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:36.150907 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:36.162091 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7ffdaeec2240dcbe4cafdb96073a7d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:36.162318 2114 raft_consensus.cc:2468] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:36.171823 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 109073eb664d4090b7a2b786146e13fa consensus state has no leader
W20251026 02:56:36.179268 2194 consensus_peers.cc:597] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:36.179615 2194 consensus_peers.cc:597] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:36.180521 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_169fb347542948c896eb9cd54f60b8ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:36.181051 2194 consensus_peers.cc:597] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4011930a277d410f863a1558416c4b0f. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:36.181170 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_169fb347542948c896eb9cd54f60b8ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.182183 2082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43402 (request call id 676) took 1222 ms. Trace:
I20251026 02:56:36.182282 2082 rpcz_store.cc:276] 1026 02:56:34.959296 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.623345 (+664049us) service_pool.cc:225] Handling call
1026 02:56:35.644933 (+ 21588us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.644959 (+ 26us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:35.807152 (+162193us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.132172 (+325020us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:36.182163 (+ 49991us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 02:56:36.182852 2194 leader_election.cc:343] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:36.183521 2194 leader_election.cc:343] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.183561 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 5226d4516ce24ecda51dab35c86da132 (table default.Table_b32c2e53d837476d9a0d5ae8ea1843d0 [id=c8435cef755d4de69c33f599df1c0f75]). Expected version 4 got 2
W20251026 02:56:36.183838 2194 consensus_peers.cc:597] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:36.184509 2194 consensus_peers.cc:597] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:36.185597 2053 consensus_peers.cc:597] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:36.186424 2378 tablet_bootstrap.cc:492] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:36.187024 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7a6f36e22379480f957ea030136b008f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:36.187763 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a6f36e22379480f957ea030136b008f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.194193 2378 tablet_bootstrap.cc:654] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.035847 2376 ts_tablet_manager.cc:1929] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:36.195349 2376 log.cc:1199] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3791cb18fe1f4937badb4859d00647f3
I20251026 02:56:36.195896 2376 ts_tablet_manager.cc:1950] T 3791cb18fe1f4937badb4859d00647f3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:36.157871 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7062c5ff91e14369ab69a2d199585c55" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:36.158172 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfc3aed64ad0439cb22fd6815c973d87" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:36.197623 3318 consensus_queue.cc:1048] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:36.198366 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 50414.4us
W20251026 02:56:36.207409 2053 leader_election.cc:343] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:36.208675 2053 leader_election.cc:343] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.209492 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 508) took 3774 ms. Trace:
I20251026 02:56:36.209657 2376 rpcz_store.cc:276] 1026 02:56:32.435393 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.435450 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:35.953007 (+3517557us) ts_tablet_manager.cc:1151] Deleting tablet 3791cb18fe1f4937badb4859d00647f3
1026 02:56:35.953016 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.032659 (+ 79643us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.035741 (+ 3082us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.209466 (+173725us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12605,"tablet-delete.queue_time_us":3517278}
I20251026 02:56:36.210121 2376 tablet_replica.cc:333] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:36.210489 2376 raft_consensus.cc:2243] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:36.211107 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3791cb18fe1f4937badb4859d00647f3 (table default.Table_9fb0008346da44318ccc90cd69474750 [id=c7a6636ca94a4863a89fbef09c339bc1]) successfully deleted
I20251026 02:56:36.212536 2376 raft_consensus.cc:2272] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.213160 2376 ts_tablet_manager.cc:1916] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:36.216681 3318 raft_consensus.cc:493] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.216827 3318 raft_consensus.cc:515] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.217576 3318 leader_election.cc:290] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:36.219828 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9391e7654c442a5bb312f0daaf1d99b consensus state has no leader
W20251026 02:56:36.220369 1916 leader_election.cc:343] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 122de8616b8f4fcf88971d5738c47e82
W20251026 02:56:36.220896 2194 consensus_peers.cc:597] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:36.221030 2194 consensus_peers.cc:597] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:36.221741 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce1ab21ddf8e45bc9063b63db9c1b97f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:36.224123 3347 tablet_replica.cc:1410] Time spent flushing the Log queue took a long time: real 0.380s user 0.008s sys 0.005s
I20251026 02:56:36.224205 2378 tablet_bootstrap.cc:492] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:36.224413 2378 ts_tablet_manager.cc:1403] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.002s
I20251026 02:56:36.225559 2378 raft_consensus.cc:359] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.225768 2378 raft_consensus.cc:385] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.225860 2378 raft_consensus.cc:740] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.226094 2378 consensus_queue.cc:260] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:36.234529 2375 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.774s user 0.003s sys 0.011s
W20251026 02:56:36.234643 2375 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.774s user 0.003s sys 0.011s
I20251026 02:56:36.235431 2375 tablet_bootstrap.cc:492] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:36.235569 2375 ts_tablet_manager.cc:1403] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.913s user 0.004s sys 0.013s
I20251026 02:56:36.241282 2377 ts_tablet_manager.cc:1929] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:36.241500 2377 log.cc:1199] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/98fe44aab33b40eba819386c3bffca46
I20251026 02:56:36.241909 2377 ts_tablet_manager.cc:1950] T 98fe44aab33b40eba819386c3bffca46 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:36.242815 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 495) took 5248 ms. Trace:
I20251026 02:56:36.243269 2378 ts_tablet_manager.cc:1434] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.019s user 0.000s sys 0.001s
I20251026 02:56:36.245113 2378 tablet_bootstrap.cc:492] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:36.246670 2378 tablet_bootstrap.cc:654] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.250090 2372 ts_tablet_manager.cc:1929] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:36.250314 2372 log.cc:1199] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/115ccccf16b74efc9f24b607cf0c74d1
I20251026 02:56:36.250936 2372 ts_tablet_manager.cc:1950] T 115ccccf16b74efc9f24b607cf0c74d1 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:36.252184 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 514) took 4602 ms. Trace:
I20251026 02:56:36.252396 2372 rpcz_store.cc:276] 1026 02:56:31.650163 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.653538 (+ 3375us) service_pool.cc:225] Handling call
1026 02:56:35.675527 (+4021989us) ts_tablet_manager.cc:1151] Deleting tablet 115ccccf16b74efc9f24b607cf0c74d1
1026 02:56:35.821754 (+146227us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.207291 (+385537us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.249962 (+ 42671us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.252161 (+ 2199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4021748}
I20251026 02:56:36.253222 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 115ccccf16b74efc9f24b607cf0c74d1 (table default.Table_7b6687267a1a4b2da7e02949e7b4c081 [id=ad981d3883cc4ee5b77defff9651ef31]) successfully deleted
I20251026 02:56:36.253929 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_833e24776bb6450a92a7b7db2e1b2481"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:36.257290 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_833e24776bb6450a92a7b7db2e1b2481 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.246266 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 865fbd674db34f668fb5a2f5836a9b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:36.263708 2377 rpcz_store.cc:276] 1026 02:56:30.994359 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:30.994420 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:36.055795 (+5061375us) ts_tablet_manager.cc:1151] Deleting tablet 98fe44aab33b40eba819386c3bffca46
1026 02:56:36.055802 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.216302 (+160500us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.241163 (+ 24861us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.242794 (+ 1631us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5061093}
I20251026 02:56:36.260993 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:36.263595 2194 leader_election.cc:343] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:36.268751 2194 consensus_peers.cc:597] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:36.269126 2194 consensus_peers.cc:597] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:36.271183 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 499) took 5140 ms. Trace:
I20251026 02:56:36.272046 2377 rpcz_store.cc:276] 1026 02:56:31.130405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.130468 (+ 63us) service_pool.cc:225] Handling call
1026 02:56:36.271073 (+5140605us) ts_tablet_manager.cc:1151] Deleting tablet 81530bbf4a2d4e908add683475e594fb
1026 02:56:36.271085 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.271167 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5140389}
I20251026 02:56:36.275601 2377 tablet_replica.cc:333] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:36.275856 2377 raft_consensus.cc:2243] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:36.276059 2377 raft_consensus.cc:2272] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.276680 2377 ts_tablet_manager.cc:1916] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:36.280588 2375 raft_consensus.cc:359] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.281055 2375 raft_consensus.cc:385] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.281184 2375 raft_consensus.cc:740] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.281558 2375 consensus_queue.cc:260] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.281575 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e586d81375544236ac7d95297fca01a8" } modify_external_catalogs: true
I20251026 02:56:36.281867 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e586d81375544236ac7d95297fca01a8" } modify_external_catalogs: true
I20251026 02:56:36.282558 2375 ts_tablet_manager.cc:1434] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.047s user 0.003s sys 0.000s
I20251026 02:56:36.282747 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet d212170f056a44dfb15cfcb7f9fbae2f from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:36.283212 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet b8a1244c1c49471fb7711a2ca9b89a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:36.249112 2381 tablet_bootstrap.cc:654] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.283640 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1361fbf89aa04321a7876c1e43e27089" } new_table_name: "default.Table_26dd741c83ff419f8c9bc808e83d9c72" modify_external_catalogs: true
W20251026 02:56:36.283797 2053 leader_election.cc:343] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.283907 2053 leader_election.cc:304] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:36.284346 2053 leader_election.cc:343] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.284439 2053 leader_election.cc:304] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:36.284791 2053 leader_election.cc:343] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.284919 2053 leader_election.cc:304] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.285125 3006 raft_consensus.cc:2749] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:36.285291 3006 raft_consensus.cc:2749] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:36.285434 3006 raft_consensus.cc:2749] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:36.285849 2053 leader_election.cc:343] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.285991 2053 leader_election.cc:304] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.286218 3006 raft_consensus.cc:2749] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:36.286533 2053 leader_election.cc:343] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.286657 2053 leader_election.cc:304] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.286870 3006 raft_consensus.cc:2749] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:36.287199 2053 leader_election.cc:343] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.287338 2053 leader_election.cc:304] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.287663 3320 raft_consensus.cc:2749] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:36.287674 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet a70d70abc3a34dac8cf9a747903fa78c (table default.Table_c4bf189953554cb8854613da705a2b42 [id=a84628b0b09f43c69e86344d40641b02]) successfully deleted
W20251026 02:56:36.287947 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 1eabfc95caa84afc9a8956a7ac13fea6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1eabfc95caa84afc9a8956a7ac13fea6 already in progress: creating tablet
W20251026 02:56:36.288084 2053 leader_election.cc:343] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:36.288180 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 4b49c0996c4d41cbaddcdf0b75846bcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b49c0996c4d41cbaddcdf0b75846bcd already in progress: creating tablet
I20251026 02:56:36.288245 2053 leader_election.cc:304] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:36.288460 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7177931353164b2aa82c00b2807ccd63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7177931353164b2aa82c00b2807ccd63 already in progress: creating tablet
I20251026 02:56:36.288512 3320 raft_consensus.cc:2749] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:36.288673 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 8fe03778a1e842829cab25c90597a3cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fe03778a1e842829cab25c90597a3cd already in progress: creating tablet
I20251026 02:56:36.290513 2378 tablet_bootstrap.cc:492] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:36.290721 2378 ts_tablet_manager.cc:1403] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.003s
I20251026 02:56:36.291476 2378 raft_consensus.cc:359] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.291718 2378 raft_consensus.cc:385] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.291817 2378 raft_consensus.cc:740] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.292057 2378 consensus_queue.cc:260] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.250095 2376 ts_tablet_manager.cc:1929] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:36.259734 3318 raft_consensus.cc:493] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.295229 3318 raft_consensus.cc:515] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.295854 3318 leader_election.cc:290] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:36.260699 1974 raft_consensus.cc:3060] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.262696 3250 tablet.cc:1722] T 740f7c7f186f45ceaa4c40e76419e3a1 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251026 02:56:36.304046 3347 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43402 (request call id 620) took 1907 ms. Trace:
I20251026 02:56:36.304189 3347 rpcz_store.cc:276] 1026 02:56:34.401714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.401803 (+ 89us) service_pool.cc:225] Handling call
1026 02:56:36.304016 (+1902213us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:34.405596 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1026 02:56:34.405692 (+ 96us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1026 02:56:34.405701 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
1026 02:56:34.405709 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1026 02:56:34.405750 (+ 41us) alter_schema_op.cc:128] START. Timestamp: 1761447394405716
1026 02:56:34.405754 (+ 4us) op_driver.cc:348] REPLICATION: starting
1026 02:56:34.405982 (+ 228us) log.cc:844] Serialized 173 byte log entry
1026 02:56:35.770900 (+1364918us) op_driver.cc:464] REPLICATION: finished
1026 02:56:35.843176 (+ 72276us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1026 02:56:35.843757 (+ 581us) log.cc:844] Serialized 5 byte log entry
1026 02:56:36.226951 (+383194us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.227255 (+ 304us) log.cc:844] Serialized 14 byte log entry
1026 02:56:36.297855 (+ 70600us) alter_schema_op.cc:73] Released schema lock
1026 02:56:36.297911 (+ 56us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":72198,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":452,"prepare.run_wall_time_us":453,"replication_time_us":1365113,"thread_start_us":459869,"threads_started":2,"wal-append.queue_time_us":364756}]]}
I20251026 02:56:36.305418 3250 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:36.305464 3347 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:36.305680 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_51689a3619574710a85dcec2830fc132" } new_table_name: "default.Table_fe3e894262cf4a89b3fcb989449d7569" modify_external_catalogs: true
I20251026 02:56:36.309973 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 98fe44aab33b40eba819386c3bffca46 (table default.Table_481575c8f05747c2954c9c79a0ebde9c [id=91d789aff2464f73b60e03b52126a396]) successfully deleted
W20251026 02:56:36.310348 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 81530bbf4a2d4e908add683475e594fb because the tablet was not found. No further retry: Not found: Tablet not found: 81530bbf4a2d4e908add683475e594fb
W20251026 02:56:36.311753 1916 leader_election.cc:343] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: aa399e6407fd4c4d9871d3d5ccf894ec
I20251026 02:56:36.312713 2113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce1ab21ddf8e45bc9063b63db9c1b97f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:36.312901 2113 raft_consensus.cc:2468] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:36.313867 2372 tablet_replica.cc:333] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:36.314074 2372 raft_consensus.cc:2243] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:36.314254 2372 raft_consensus.cc:2272] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.314904 2372 ts_tablet_manager.cc:1916] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:36.322914 2375 tablet_bootstrap.cc:492] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:36.324862 2089 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43402 (request call id 652) took 1369 ms. Trace:
I20251026 02:56:36.325032 2089 rpcz_store.cc:276] 1026 02:56:34.955675 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.955718 (+ 43us) service_pool.cc:225] Handling call
1026 02:56:35.193566 (+237848us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.193587 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:36.015865 (+822278us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.286114 (+270249us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:36.324844 (+ 38730us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:36.326251 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c8a0cf8b6d464c81825ff3b2d17afeaf" } modify_external_catalogs: true
I20251026 02:56:36.326568 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c8a0cf8b6d464c81825ff3b2d17afeaf" } modify_external_catalogs: true
I20251026 02:56:36.327445 2375 tablet_bootstrap.cc:654] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.333954 2376 log.cc:1199] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/84ad6c72a8704c148411881760848df4
I20251026 02:56:36.334463 2376 ts_tablet_manager.cc:1950] T 84ad6c72a8704c148411881760848df4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:36.334821 3328 consensus_queue.cc:1048] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:36.335848 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 509) took 3780 ms. Trace:
I20251026 02:56:36.335989 2376 rpcz_store.cc:276] 1026 02:56:32.555160 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.555377 (+ 217us) service_pool.cc:225] Handling call
1026 02:56:36.210071 (+3654694us) ts_tablet_manager.cc:1151] Deleting tablet 84ad6c72a8704c148411881760848df4
1026 02:56:36.210083 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.246795 (+ 36712us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.250018 (+ 3223us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.335826 (+ 85808us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3654456}
I20251026 02:56:36.336472 2376 tablet_replica.cc:333] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:36.336656 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 84ad6c72a8704c148411881760848df4 (table default.Table_7814874203fc40af9ef07eeb2e11ebc4 [id=cc22b2daa8fb4eb1bae52b189d66b8fb]) successfully deleted
I20251026 02:56:36.344914 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d08c9c1fe775483492716153dba0072d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:36.345342 3006 raft_consensus.cc:493] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.345503 3006 raft_consensus.cc:515] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:36.345705 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d08c9c1fe775483492716153dba0072d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.346195 3006 leader_election.cc:290] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:36.346777 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7801f073d5d94076af9299492f86fad2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:36.347342 2053 leader_election.cc:343] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.351193 2089 tablet_service.cc:1505] Processing CreateTablet for tablet 6dbe3f1f3e074da5bae9badcffa01f6b (DEFAULT_TABLE table=default.Table_000637f9a96344c28b90e88c10c89a07 [id=c788731642224bcf9c22c007e8ab35c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.351579 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dbe3f1f3e074da5bae9badcffa01f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.351648 2241 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35450 (request call id 667) took 1548 ms. Trace:
I20251026 02:56:36.351725 2241 rpcz_store.cc:276] 1026 02:56:34.802951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.128716 (+325765us) service_pool.cc:225] Handling call
1026 02:56:35.200663 (+ 71947us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1026 02:56:35.200686 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1026 02:56:36.277998 (+1077312us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.351462 (+ 73464us) tablet_replica.cc:206] Creating consensus instance
1026 02:56:36.351630 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:36.355327 3006 raft_consensus.cc:493] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.355443 3006 raft_consensus.cc:515] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.355973 3006 leader_election.cc:290] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:36.356563 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d2cc6b16d34331bb80c576c0cd2ce0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:36.356523 1660 catalog_manager.cc:5649] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } }
W20251026 02:56:36.357213 2053 leader_election.cc:343] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.357359 1660 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 352a284902c94f088e2e6ae006162db0 (table default.Table_6cd4ee82e9ec406f971669b7f0722677 [id=414375003cd44b5c8c9f0249b339c28a]). Expected version 1 got 0
I20251026 02:56:36.357553 2195 leader_election.cc:304] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.357824 3239 raft_consensus.cc:2804] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:36.357914 3239 raft_consensus.cc:493] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:36.357981 3239 raft_consensus.cc:3060] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.358562 2381 tablet_bootstrap.cc:492] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:36.358672 2381 ts_tablet_manager.cc:1403] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.352s user 0.004s sys 0.000s
I20251026 02:56:36.359274 2381 raft_consensus.cc:359] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.359432 2381 raft_consensus.cc:385] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.359479 2381 raft_consensus.cc:740] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.359766 2381 consensus_queue.cc:260] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.360345 2381 ts_tablet_manager.cc:1434] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:36.360698 2381 tablet_bootstrap.cc:492] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:36.360816 3239 raft_consensus.cc:515] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.361217 1947 tablet_service.cc:1505] Processing CreateTablet for tablet 6dbe3f1f3e074da5bae9badcffa01f6b (DEFAULT_TABLE table=default.Table_000637f9a96344c28b90e88c10c89a07 [id=c788731642224bcf9c22c007e8ab35c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.361389 3239 leader_election.cc:290] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:36.361730 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dbe3f1f3e074da5bae9badcffa01f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.362282 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e9b0e1d3f3f4fdbb8c58f26a4308771" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:36.363317 2194 leader_election.cc:343] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.364626 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 2d8c9c0792f944758fbaa658a99f1c71 (DEFAULT_TABLE table=default.Table_8300040c56d34cf985a2009545c1eeb5 [id=8e2a88320e9a49aabf9e560c6861343b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.365080 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d8c9c0792f944758fbaa658a99f1c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.368661 2378 ts_tablet_manager.cc:1434] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.078s user 0.001s sys 0.001s
W20251026 02:56:36.369900 2195 consensus_peers.cc:597] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a111848706b743c9beb2fbd21f59d7f3. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:36.371219 2378 tablet_bootstrap.cc:492] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:36.372814 2378 tablet_bootstrap.cc:654] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.374071 1954 tablet_service.cc:1505] Processing CreateTablet for tablet bfc5a4c9394d49879eb2cedb480452ae (DEFAULT_TABLE table=default.Table_d4465a7252ec4e43ac0819aab10e15d0 [id=a94b5d2ed09d4372951c41fb30cb95ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.374593 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfc5a4c9394d49879eb2cedb480452ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.375913 2376 raft_consensus.cc:2243] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:36.376072 2376 raft_consensus.cc:2272] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.376607 2376 ts_tablet_manager.cc:1916] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:36.380800 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 2d8c9c0792f944758fbaa658a99f1c71 (DEFAULT_TABLE table=default.Table_8300040c56d34cf985a2009545c1eeb5 [id=8e2a88320e9a49aabf9e560c6861343b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.381009 3328 consensus_queue.cc:1048] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:36.382009 2195 leader_election.cc:304] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.382269 2962 raft_consensus.cc:2804] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:36.382347 2962 raft_consensus.cc:493] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:36.382401 2962 raft_consensus.cc:3060] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.385499 2962 raft_consensus.cc:515] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.386123 2962 leader_election.cc:290] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:36.388414 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7ffdaeec2240dcbe4cafdb96073a7d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:36.388978 2194 leader_election.cc:343] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:36.390934 2195 consensus_peers.cc:597] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 276bd3b97c97426daf802e1f57feb265. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:36.391536 3328 consensus_queue.cc:1048] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:36.392427 2195 leader_election.cc:304] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.392691 3239 raft_consensus.cc:2804] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:36.392778 3239 raft_consensus.cc:493] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:36.392855 3239 raft_consensus.cc:3060] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.395567 3239 raft_consensus.cc:515] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.398171 2095 tablet_service.cc:1505] Processing CreateTablet for tablet bfc5a4c9394d49879eb2cedb480452ae (DEFAULT_TABLE table=default.Table_d4465a7252ec4e43ac0819aab10e15d0 [id=a94b5d2ed09d4372951c41fb30cb95ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.399227 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 863aef239cd34a358a71ddfdba6fb72a (DEFAULT_TABLE table=default.Table_70b9f6803b9345bc8010f0e216bff485 [id=b7887cd8af7c4ef69653751977a9707b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.399736 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 863aef239cd34a358a71ddfdba6fb72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.402158 2377 ts_tablet_manager.cc:1929] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:36.402269 2377 log.cc:1199] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/b1a2522801364693be0bda6fa45a697e
I20251026 02:56:36.406634 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 863aef239cd34a358a71ddfdba6fb72a (DEFAULT_TABLE table=default.Table_70b9f6803b9345bc8010f0e216bff485 [id=b7887cd8af7c4ef69653751977a9707b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.411514 2381 tablet_bootstrap.cc:654] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.413877 3239 leader_election.cc:290] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:36.414182 3347 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43402 (request call id 685) took 1100 ms. Trace:
I20251026 02:56:36.414284 3347 rpcz_store.cc:276] 1026 02:56:35.313237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.641192 (+327955us) service_pool.cc:225] Handling call
1026 02:56:36.414162 (+772970us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:36.048401 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1026 02:56:36.048563 (+ 162us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1026 02:56:36.297963 (+249400us) alter_schema_op.cc:66] Acquired schema lock
1026 02:56:36.297972 (+ 9us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1026 02:56:36.298009 (+ 37us) alter_schema_op.cc:128] START. Timestamp: 1761447396297951
1026 02:56:36.298013 (+ 4us) op_driver.cc:348] REPLICATION: starting
1026 02:56:36.298290 (+ 277us) log.cc:844] Serialized 173 byte log entry
1026 02:56:36.302249 (+ 3959us) op_driver.cc:464] REPLICATION: finished
1026 02:56:36.305431 (+ 3182us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1026 02:56:36.305858 (+ 427us) log.cc:844] Serialized 5 byte log entry
1026 02:56:36.343979 (+ 38121us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.402592 (+ 58613us) log.cc:844] Serialized 14 byte log entry
1026 02:56:36.414105 (+ 11513us) alter_schema_op.cc:73] Released schema lock
1026 02:56:36.414112 (+ 7us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3133,"prepare.queue_time_us":406837,"prepare.run_cpu_time_us":4420,"prepare.run_wall_time_us":250432,"raft.queue_time_us":66,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":229,"replication_time_us":4205,"thread_start_us":309945,"threads_started":2}]]}
I20251026 02:56:36.415652 2378 tablet_bootstrap.cc:492] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:36.415863 2378 ts_tablet_manager.cc:1403] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20251026 02:56:36.416769 2378 raft_consensus.cc:359] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.417037 2378 raft_consensus.cc:385] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.417157 2378 raft_consensus.cc:740] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.417464 2378 consensus_queue.cc:260] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.418943 2375 tablet_bootstrap.cc:492] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:36.418989 3328 raft_consensus.cc:493] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.419091 2375 ts_tablet_manager.cc:1403] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.096s user 0.001s sys 0.002s
I20251026 02:56:36.419246 3328 raft_consensus.cc:515] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.419874 3328 leader_election.cc:290] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:36.419811 2375 raft_consensus.cc:359] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.419968 2375 raft_consensus.cc:385] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.420027 2375 raft_consensus.cc:740] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.420308 2375 consensus_queue.cc:260] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.420581 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ced7ad6a601d432cbc13b3199a0506e4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:36.420735 1978 raft_consensus.cc:2468] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:36.420780 2375 ts_tablet_manager.cc:1434] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:36.421046 2375 tablet_bootstrap.cc:492] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:36.422297 2194 leader_election.cc:304] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:36.423038 3328 raft_consensus.cc:2804] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:36.423182 3328 raft_consensus.cc:493] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:36.423305 3328 raft_consensus.cc:3060] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.426339 3328 raft_consensus.cc:515] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.427479 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ced7ad6a601d432cbc13b3199a0506e4" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:36.427646 1978 raft_consensus.cc:3060] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.427686 2375 tablet_bootstrap.cc:654] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.428427 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce1ab21ddf8e45bc9063b63db9c1b97f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:36.430855 1978 raft_consensus.cc:2468] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:36.431385 2194 leader_election.cc:343] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.431689 2194 leader_election.cc:304] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:36.434386 3239 raft_consensus.cc:2804] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:36.434788 3239 raft_consensus.cc:697] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:36.435055 3239 consensus_queue.cc:237] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.438333 2376 ts_tablet_manager.cc:1929] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:36.438524 2376 log.cc:1199] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/aa399e6407fd4c4d9871d3d5ccf894ec
I20251026 02:56:36.439059 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfc5a4c9394d49879eb2cedb480452ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.439400 2376 ts_tablet_manager.cc:1950] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:36.439657 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 863aef239cd34a358a71ddfdba6fb72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.439851 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 510) took 3884 ms. Trace:
I20251026 02:56:36.439926 2376 rpcz_store.cc:276] 1026 02:56:32.555318 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.555652 (+ 334us) service_pool.cc:225] Handling call
1026 02:56:36.336425 (+3780773us) ts_tablet_manager.cc:1151] Deleting tablet aa399e6407fd4c4d9871d3d5ccf894ec
1026 02:56:36.336437 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.379575 (+ 43138us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.438207 (+ 58632us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.439832 (+ 1625us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39180,"tablet-delete.queue_time_us":3780612}
I20251026 02:56:36.446166 2377 ts_tablet_manager.cc:1950] T b1a2522801364693be0bda6fa45a697e P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:36.447093 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 498) took 5316 ms. Trace:
I20251026 02:56:36.447268 2377 rpcz_store.cc:276] 1026 02:56:31.130286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.130702 (+ 416us) service_pool.cc:225] Handling call
1026 02:56:36.275548 (+5144846us) ts_tablet_manager.cc:1151] Deleting tablet b1a2522801364693be0bda6fa45a697e
1026 02:56:36.275559 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.280793 (+ 5234us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.402031 (+121238us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.447068 (+ 45037us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5144677}
I20251026 02:56:36.447810 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 500) took 5317 ms. Trace:
I20251026 02:56:36.448581 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d8c9c0792f944758fbaa658a99f1c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.448580 2377 rpcz_store.cc:276] 1026 02:56:31.130526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.130898 (+ 372us) service_pool.cc:225] Handling call
1026 02:56:36.447689 (+5316791us) ts_tablet_manager.cc:1151] Deleting tablet c570e5f9a93c45bda32ec92c68794714
1026 02:56:36.447700 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.447792 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5315913}
I20251026 02:56:36.449038 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 501) took 5318 ms. Trace:
I20251026 02:56:36.453521 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7ece114e4cfd4133871448f9ea3ce9e2" } modify_external_catalogs: true
I20251026 02:56:36.453822 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7ece114e4cfd4133871448f9ea3ce9e2" } modify_external_catalogs: true
I20251026 02:56:36.461083 2089 tablet_service.cc:1505] Processing CreateTablet for tablet 572afa86823247a8b1744813350c8eff (DEFAULT_TABLE table=default.Table_a036080d1e7141269eec20b1f8e87f9c [id=59014691dadb4367900e21e182038d3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.461591 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572afa86823247a8b1744813350c8eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.465998 2377 rpcz_store.cc:276] 1026 02:56:31.130651 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.131805 (+ 1154us) service_pool.cc:225] Handling call
1026 02:56:36.448947 (+5317142us) ts_tablet_manager.cc:1151] Deleting tablet 4a5343ba771a4e8ea2f467c4c619c1f9
1026 02:56:36.448954 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.449020 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5316957}
I20251026 02:56:36.466455 2372 ts_tablet_manager.cc:1929] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:36.466574 2372 log.cc:1199] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/1e56b259a314422aba35eb129021bbeb
I20251026 02:56:36.466980 2372 ts_tablet_manager.cc:1950] T 1e56b259a314422aba35eb129021bbeb P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:36.467763 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 515) took 4817 ms. Trace:
I20251026 02:56:36.467851 2372 rpcz_store.cc:276] 1026 02:56:31.650313 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.655132 (+ 4819us) service_pool.cc:225] Handling call
1026 02:56:36.253588 (+4598456us) ts_tablet_manager.cc:1151] Deleting tablet 1e56b259a314422aba35eb129021bbeb
1026 02:56:36.253600 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.373389 (+119789us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.466339 (+ 92950us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.467741 (+ 1402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4595135}
I20251026 02:56:36.468288 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 1e56b259a314422aba35eb129021bbeb (table default.Table_25607539f64440cb8ff2c24a0a3a3bdd [id=5dbfb5d912ba433baa46bd52bbf6dbee]) successfully deleted
I20251026 02:56:36.468720 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 517) took 4687 ms. Trace:
I20251026 02:56:36.468852 2372 rpcz_store.cc:276] 1026 02:56:31.781277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.808591 (+ 27314us) service_pool.cc:225] Handling call
1026 02:56:36.468589 (+4659998us) ts_tablet_manager.cc:1151] Deleting tablet c31d2704d1f14ddead69c794cc631ec1
1026 02:56:36.468600 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.468701 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4659674}
W20251026 02:56:36.465854 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 50704.6us
I20251026 02:56:36.469973 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfc3aed64ad0439cb22fd6815c973d87" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:36.470183 2260 raft_consensus.cc:2468] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:36.465714 3328 leader_election.cc:290] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:36.470465 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7062c5ff91e14369ab69a2d199585c55" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:36.470613 2262 raft_consensus.cc:2468] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
W20251026 02:56:36.471614 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b819b43894574dc9ad52b4081e46628e consensus state has no leader
I20251026 02:56:36.471752 2254 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7801f073d5d94076af9299492f86fad2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:36.472048 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d2cc6b16d34331bb80c576c0cd2ce0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:36.483608 2377 tablet_replica.cc:333] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:36.486109 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cef54300186c47eca175dff1adf385db" } modify_external_catalogs: true
I20251026 02:56:36.486387 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cef54300186c47eca175dff1adf385db" } modify_external_catalogs: true
I20251026 02:56:36.488487 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet aa399e6407fd4c4d9871d3d5ccf894ec from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:36.489197 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6a273a7631fc410e8ba77d604a7e1c66" } new_table_name: "default.Table_74da7d358fc34e3ab4ad068d26ab8f01" modify_external_catalogs: true
I20251026 02:56:36.491573 2377 raft_consensus.cc:2243] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:36.492060 2377 raft_consensus.cc:2272] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.492795 2377 ts_tablet_manager.cc:1916] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:36.493980 1648 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 193e32898b164133b954409a08c298ce (table default.Table_6193b9cb563841bcbf42d20cf7a21f59 [id=52c2da2abb1e4417bd60703a5f672e70], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 193e32898b164133b954409a08c298ce consensus state has no leader
I20251026 02:56:36.494436 2381 tablet_bootstrap.cc:492] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:36.494567 2381 ts_tablet_manager.cc:1403] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.134s user 0.003s sys 0.000s
I20251026 02:56:36.495255 2381 raft_consensus.cc:359] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.495448 2381 raft_consensus.cc:385] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.495494 2381 raft_consensus.cc:740] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.495716 2381 consensus_queue.cc:260] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.496441 2381 ts_tablet_manager.cc:1434] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:36.496762 2381 tablet_bootstrap.cc:492] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:36.498283 2381 tablet_bootstrap.cc:654] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:36.500432 1629 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 44969.7us
I20251026 02:56:36.512439 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1c740ab6481e4481b95878c3a185129d
I20251026 02:56:36.513478 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_103c3d38235f4771a4cad7b79143a809" } new_table_name: "default.Table_2cec699a24664381804fc628fb6ebcc9" modify_external_catalogs: true
I20251026 02:56:36.515230 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 64415f5588db4df0a94676b650fb3631
I20251026 02:56:36.515739 1661 catalog_manager.cc:6719] ReplaceTablet: tablet 64415f5588db4df0a94676b650fb3631 deleted and replaced by tablet d0b0954ff4824ee7b47dce18bdc61122
I20251026 02:56:36.516196 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1f67f077a0ce4edab1c2711aa4d9556c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:36.516963 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f67f077a0ce4edab1c2711aa4d9556c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.481868 2095 tablet_service.cc:1505] Processing CreateTablet for tablet e0e0d8ea9cb04f97814ca4d6075429f6 (DEFAULT_TABLE table=default.Table_128d37058db744249e0cd066307fbe22 [id=854e575d29d44ab59f4955b85bb1c3f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.521847 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e0d8ea9cb04f97814ca4d6075429f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.485234 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 94bdaa97e7604040b7fc94fe22e743bf (DEFAULT_TABLE table=default.Table_c119e6aabe874f9495d15c66088c7440 [id=bf4ebfe2d12a476689c255ecdcb169bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.523458 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94bdaa97e7604040b7fc94fe22e743bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.485472 2099 tablet_service.cc:1505] Processing CreateTablet for tablet b819b43894574dc9ad52b4081e46628e (DEFAULT_TABLE table=default.Table_fe3e894262cf4a89b3fcb989449d7569 [id=bb813553ce6b4ed587777838fc90508d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.500445 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e9b0e1d3f3f4fdbb8c58f26a4308771" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:36.519272 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c740ab6481e4481b95878c3a185129d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.519448 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 64415f5588db4df0a94676b650fb3631 with delete_type TABLET_DATA_DELETED (replaced by tablet d0b0954ff4824ee7b47dce18bdc61122) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.524164 2120 raft_consensus.cc:3060] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.526753 2120 raft_consensus.cc:2468] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:36.527387 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b819b43894574dc9ad52b4081e46628e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.537052 2372 tablet_replica.cc:333] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:36.537353 2372 raft_consensus.cc:2243] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:36.537559 2372 raft_consensus.cc:2272] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.538129 2372 ts_tablet_manager.cc:1916] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:36.540766 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7ffdaeec2240dcbe4cafdb96073a7d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:36.540958 2120 raft_consensus.cc:3060] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.542104 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 0ddeba351c264f67b75b66c55c9734b5 (DEFAULT_TABLE table=default.Table_0207b8f2a960407fa3a85a1280bb2256 [id=03d68f4637214d9e81939251031d9372]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.542532 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ddeba351c264f67b75b66c55c9734b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.543403 2120 raft_consensus.cc:2468] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:36.450326 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 572afa86823247a8b1744813350c8eff (DEFAULT_TABLE table=default.Table_a036080d1e7141269eec20b1f8e87f9c [id=59014691dadb4367900e21e182038d3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.544104 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572afa86823247a8b1744813350c8eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.548542 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet aa399e6407fd4c4d9871d3d5ccf894ec (table default.Table_65488c1f07e542b5973be5cfd122787c [id=ce3610a9d6814ec783ae7503e58ef3ad]) successfully deleted
I20251026 02:56:36.548877 2376 tablet_replica.cc:333] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:36.452559 1947 tablet_service.cc:1505] Processing CreateTablet for tablet 94bdaa97e7604040b7fc94fe22e743bf (DEFAULT_TABLE table=default.Table_c119e6aabe874f9495d15c66088c7440 [id=bf4ebfe2d12a476689c255ecdcb169bf]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:36.471443 1915 consensus_peers.cc:597] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Couldn't send request to peer 696476265495492790739ec126a08539. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:36.549124 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 77723.4us
I20251026 02:56:36.549171 2376 raft_consensus.cc:2243] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:36.549365 2376 raft_consensus.cc:2272] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:36.549428 1915 consensus_peers.cc:597] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Couldn't send request to peer 696476265495492790739ec126a08539. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a70d70abc3a34dac8cf9a747903fa78c. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:36.549597 1915 leader_election.cc:343] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 122de8616b8f4fcf88971d5738c47e82
I20251026 02:56:36.549705 1915 leader_election.cc:304] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:36.549980 2376 ts_tablet_manager.cc:1916] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:36.550172 1915 leader_election.cc:343] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: aa399e6407fd4c4d9871d3d5ccf894ec
I20251026 02:56:36.550275 1915 leader_election.cc:304] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
W20251026 02:56:36.550480 1915 raft_consensus.cc:2686] T aa399e6407fd4c4d9871d3d5ccf894ec P 7fef01f4f2b843538ea245a09dd087a8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251026 02:56:36.551457 3318 raft_consensus.cc:2749] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:36.552647 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94bdaa97e7604040b7fc94fe22e743bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.509599 3401 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:36.481132 1960 tablet_service.cc:1505] Processing CreateTablet for tablet e0e0d8ea9cb04f97814ca4d6075429f6 (DEFAULT_TABLE table=default.Table_128d37058db744249e0cd066307fbe22 [id=854e575d29d44ab59f4955b85bb1c3f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.483759 1954 tablet_service.cc:1505] Processing CreateTablet for tablet b819b43894574dc9ad52b4081e46628e (DEFAULT_TABLE table=default.Table_fe3e894262cf4a89b3fcb989449d7569 [id=bb813553ce6b4ed587777838fc90508d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.557760 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b819b43894574dc9ad52b4081e46628e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:36.495209 1916 consensus_peers.cc:597] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a70d70abc3a34dac8cf9a747903fa78c. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:36.558836 1916 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 63663.6us
I20251026 02:56:36.559245 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce1ab21ddf8e45bc9063b63db9c1b97f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:36.559420 2111 raft_consensus.cc:3060] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.509436 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 0ddeba351c264f67b75b66c55c9734b5 (DEFAULT_TABLE table=default.Table_0207b8f2a960407fa3a85a1280bb2256 [id=03d68f4637214d9e81939251031d9372]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.559784 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ddeba351c264f67b75b66c55c9734b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.562023 2111 raft_consensus.cc:2468] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:36.562605 2375 tablet_bootstrap.cc:492] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:36.562736 2375 ts_tablet_manager.cc:1403] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.142s user 0.000s sys 0.005s
I20251026 02:56:36.520659 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c740ab6481e4481b95878c3a185129d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:36.524433 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 64415f5588db4df0a94676b650fb3631 with delete_type TABLET_DATA_DELETED (replaced by tablet d0b0954ff4824ee7b47dce18bdc61122) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:36.563802 2375 raft_consensus.cc:359] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.564000 2375 raft_consensus.cc:385] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.564057 2375 raft_consensus.cc:740] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.533262 1978 raft_consensus.cc:3060] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.452533 2378 ts_tablet_manager.cc:1434] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20251026 02:56:36.564338 2375 consensus_queue.cc:260] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.564961 2378 tablet_bootstrap.cc:492] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:36.565642 2375 ts_tablet_manager.cc:1434] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251026 02:56:36.566502 2375 tablet_bootstrap.cc:492] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:36.566735 2378 tablet_bootstrap.cc:654] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.568392 2375 tablet_bootstrap.cc:654] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.570376 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet b1a2522801364693be0bda6fa45a697e (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]) successfully deleted
W20251026 02:56:36.570873 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet c570e5f9a93c45bda32ec92c68794714 because the tablet was not found. No further retry: Not found: Tablet not found: c570e5f9a93c45bda32ec92c68794714
W20251026 02:56:36.572070 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 4a5343ba771a4e8ea2f467c4c619c1f9 because the tablet was not found. No further retry: Not found: Tablet not found: 4a5343ba771a4e8ea2f467c4c619c1f9
I20251026 02:56:36.573182 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 6dbe3f1f3e074da5bae9badcffa01f6b (DEFAULT_TABLE table=default.Table_000637f9a96344c28b90e88c10c89a07 [id=c788731642224bcf9c22c007e8ab35c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.573583 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dbe3f1f3e074da5bae9badcffa01f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.583662 3318 raft_consensus.cc:493] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.583824 3318 raft_consensus.cc:515] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.584437 3318 leader_election.cc:290] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:36.585223 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6d6f024d8b943d5acf69b64bc7c7214" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:36.587081 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e0d8ea9cb04f97814ca4d6075429f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.597681 2372 ts_tablet_manager.cc:1929] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:36.597936 2372 log.cc:1199] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/4011930a277d410f863a1558416c4b0f
I20251026 02:56:36.598390 2372 ts_tablet_manager.cc:1950] T 4011930a277d410f863a1558416c4b0f P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:36.599264 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 518) took 4736 ms. Trace:
I20251026 02:56:36.600281 2372 rpcz_store.cc:276] 1026 02:56:31.863053 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.896403 (+ 33350us) service_pool.cc:225] Handling call
1026 02:56:36.537007 (+4640604us) ts_tablet_manager.cc:1151] Deleting tablet 4011930a277d410f863a1558416c4b0f
1026 02:56:36.537014 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.580598 (+ 43584us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.583680 (+ 3082us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.599245 (+ 15565us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4640075}
I20251026 02:56:36.600186 2375 tablet_bootstrap.cc:492] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:36.600735 2375 ts_tablet_manager.cc:1403] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.003s
I20251026 02:56:36.600836 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 2d8c9c0792f944758fbaa658a99f1c71 (DEFAULT_TABLE table=default.Table_8300040c56d34cf985a2009545c1eeb5 [id=8e2a88320e9a49aabf9e560c6861343b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.601553 2375 raft_consensus.cc:359] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.601716 2375 raft_consensus.cc:385] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.601763 2375 raft_consensus.cc:740] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.602003 2375 consensus_queue.cc:260] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.603322 2375 ts_tablet_manager.cc:1434] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 02:56:36.603605 2375 tablet_bootstrap.cc:492] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:36.605126 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 4011930a277d410f863a1558416c4b0f (table default.Table_8dcca91177f5452e816e2080bceadb93 [id=30cb7a766d854568892ad3a733d34f49]) successfully deleted
I20251026 02:56:36.605346 2226 tablet_service.cc:1505] Processing CreateTablet for tablet bfc5a4c9394d49879eb2cedb480452ae (DEFAULT_TABLE table=default.Table_d4465a7252ec4e43ac0819aab10e15d0 [id=a94b5d2ed09d4372951c41fb30cb95ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.605600 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 523) took 4434 ms. Trace:
I20251026 02:56:36.605671 2372 rpcz_store.cc:276] 1026 02:56:32.171150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.171202 (+ 52us) service_pool.cc:225] Handling call
1026 02:56:36.605476 (+4434274us) ts_tablet_manager.cc:1151] Deleting tablet 9fb42b54f558499090a1d7cf257dd84c
1026 02:56:36.605486 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.605582 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4434000}
I20251026 02:56:36.606240 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 863aef239cd34a358a71ddfdba6fb72a (DEFAULT_TABLE table=default.Table_70b9f6803b9345bc8010f0e216bff485 [id=b7887cd8af7c4ef69653751977a9707b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.606541 2230 tablet_service.cc:1505] Processing CreateTablet for tablet e0e0d8ea9cb04f97814ca4d6075429f6 (DEFAULT_TABLE table=default.Table_128d37058db744249e0cd066307fbe22 [id=854e575d29d44ab59f4955b85bb1c3f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.606664 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 572afa86823247a8b1744813350c8eff (DEFAULT_TABLE table=default.Table_a036080d1e7141269eec20b1f8e87f9c [id=59014691dadb4367900e21e182038d3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.607227 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet c13d89fc829049428363ede24c350ee5
I20251026 02:56:36.607398 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 94bdaa97e7604040b7fc94fe22e743bf (DEFAULT_TABLE table=default.Table_c119e6aabe874f9495d15c66088c7440 [id=bf4ebfe2d12a476689c255ecdcb169bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.607455 2053 leader_election.cc:304] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.607828 3371 raft_consensus.cc:2804] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:36.607802 2224 tablet_service.cc:1505] Processing CreateTablet for tablet b819b43894574dc9ad52b4081e46628e (DEFAULT_TABLE table=default.Table_fe3e894262cf4a89b3fcb989449d7569 [id=bb813553ce6b4ed587777838fc90508d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.607893 3371 raft_consensus.cc:493] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:36.607947 3371 raft_consensus.cc:3060] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.608106 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c740ab6481e4481b95878c3a185129d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:36.608309 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 64415f5588db4df0a94676b650fb3631 with delete_type TABLET_DATA_DELETED (replaced by tablet d0b0954ff4824ee7b47dce18bdc61122) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:36.609225 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9fb42b54f558499090a1d7cf257dd84c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fb42b54f558499090a1d7cf257dd84c already in progress: creating tablet
I20251026 02:56:36.609264 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f0cdd3958d9d4428954284d993884629"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:36.610029 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0cdd3958d9d4428954284d993884629 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.610060 2243 tablet_service.cc:1505] Processing CreateTablet for tablet 0ddeba351c264f67b75b66c55c9734b5 (DEFAULT_TABLE table=default.Table_0207b8f2a960407fa3a85a1280bb2256 [id=03d68f4637214d9e81939251031d9372]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:36.610893 3371 raft_consensus.cc:515] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.611891 3371 leader_election.cc:290] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:36.612632 2053 leader_election.cc:304] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.612818 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfc3aed64ad0439cb22fd6815c973d87" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:36.613091 3371 raft_consensus.cc:2804] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:36.613190 3371 raft_consensus.cc:493] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:36.613281 3371 raft_consensus.cc:3060] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:36.613830 2053 leader_election.cc:343] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.614071 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet c13d89fc829049428363ede24c350ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.614419 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet c13d89fc829049428363ede24c350ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:36.617034 3239 raft_consensus.cc:493] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.617189 3239 raft_consensus.cc:515] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.617816 3239 leader_election.cc:290] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:36.617942 3371 raft_consensus.cc:515] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.618497 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce922040298d4952af13652eeda7a234" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:36.618781 3371 leader_election.cc:290] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:36.619046 2194 leader_election.cc:343] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.619521 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet f719c720cc474181a63786a3e248ab2c
I20251026 02:56:36.619581 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7062c5ff91e14369ab69a2d199585c55" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:36.620003 2053 leader_election.cc:343] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.620426 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 3327d26a2c2c4d0b86d98151f5940bd7 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:36.620472 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 524) took 4446 ms. Trace:
I20251026 02:56:36.620539 2372 rpcz_store.cc:276] 1026 02:56:32.174172 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.175780 (+ 1608us) service_pool.cc:225] Handling call
1026 02:56:36.620345 (+4444565us) ts_tablet_manager.cc:1151] Deleting tablet 21b936fa4f6a4ff6af321971a30cccbe
1026 02:56:36.620357 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.620457 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4444338}
I20251026 02:56:36.621167 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_81e9f54ed3934b6b80f7c10d0be54d9a" } new_table_name: "default.Table_3a75d3c1b8304d1bbcc19c12111f24dd" modify_external_catalogs: true
I20251026 02:56:36.622898 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 525) took 4444 ms. Trace:
I20251026 02:56:36.622970 2372 rpcz_store.cc:276] 1026 02:56:32.177997 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.178202 (+ 205us) service_pool.cc:225] Handling call
1026 02:56:36.622784 (+4444582us) ts_tablet_manager.cc:1151] Deleting tablet 027dd92fc5cf4c9e847de3e55e0dac1c
1026 02:56:36.622795 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.622881 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4444391}
W20251026 02:56:36.623556 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 21b936fa4f6a4ff6af321971a30cccbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21b936fa4f6a4ff6af321971a30cccbe already in progress: creating tablet
I20251026 02:56:36.623739 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 526) took 4445 ms. Trace:
I20251026 02:56:36.623801 2372 rpcz_store.cc:276] 1026 02:56:32.178159 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.178329 (+ 170us) service_pool.cc:225] Handling call
1026 02:56:36.623648 (+4445319us) ts_tablet_manager.cc:1151] Deleting tablet 4a3f3fc3c7a04d05bae28472251f3acc
1026 02:56:36.623656 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.623725 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4445136}
I20251026 02:56:36.624007 2372 tablet_replica.cc:333] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:36.624382 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet f719c720cc474181a63786a3e248ab2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:36.624606 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 027dd92fc5cf4c9e847de3e55e0dac1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 027dd92fc5cf4c9e847de3e55e0dac1c already in progress: creating tablet
W20251026 02:56:36.624923 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 4a3f3fc3c7a04d05bae28472251f3acc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a3f3fc3c7a04d05bae28472251f3acc already in progress: creating tablet
I20251026 02:56:36.625284 2372 raft_consensus.cc:2243] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:36.627197 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet f719c720cc474181a63786a3e248ab2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:36.627842 2372 raft_consensus.cc:2272] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.629154 2372 ts_tablet_manager.cc:1916] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:36.630228 2378 tablet_bootstrap.cc:492] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:36.630365 2378 ts_tablet_manager.cc:1403] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.003s
I20251026 02:56:36.631063 2378 raft_consensus.cc:359] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.631258 2378 raft_consensus.cc:385] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.631320 2378 raft_consensus.cc:740] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.631552 2378 consensus_queue.cc:260] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.640161 3394 tablet.cc:1722] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20251026 02:56:36.641681 2053 leader_election.cc:343] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.641809 2053 leader_election.cc:304] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:36.642313 2053 leader_election.cc:343] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.642400 2053 leader_election.cc:304] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.642589 3006 raft_consensus.cc:2749] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:36.642729 3006 raft_consensus.cc:2749] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:36.643337 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfc5a4c9394d49879eb2cedb480452ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.643748 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 863aef239cd34a358a71ddfdba6fb72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.643944 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b819b43894574dc9ad52b4081e46628e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.644309 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ddeba351c264f67b75b66c55c9734b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.644646 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94bdaa97e7604040b7fc94fe22e743bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.644965 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572afa86823247a8b1744813350c8eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.645136 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e0d8ea9cb04f97814ca4d6075429f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.648084 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d8c9c0792f944758fbaa658a99f1c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:36.650247 3318 raft_consensus.cc:493] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.650414 3318 raft_consensus.cc:515] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.651067 3318 leader_election.cc:290] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:36.655151 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2bfd9b55c484ef8a9c6345d56f794fb" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:36.656391 1974 raft_consensus.cc:3060] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.657984 2258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6d6f024d8b943d5acf69b64bc7c7214" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:36.658363 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2bfd9b55c484ef8a9c6345d56f794fb" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:36.662568 2378 ts_tablet_manager.cc:1434] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20251026 02:56:36.663709 2378 tablet_bootstrap.cc:492] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:36.665330 2378 tablet_bootstrap.cc:654] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.684100 2375 tablet_bootstrap.cc:654] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.689764 2376 ts_tablet_manager.cc:1929] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:36.692687 2375 tablet_bootstrap.cc:492] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:36.692962 2375 ts_tablet_manager.cc:1403] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.002s
I20251026 02:56:36.698208 2372 ts_tablet_manager.cc:1929] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:36.701509 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfc3aed64ad0439cb22fd6815c973d87" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:36.701709 2257 raft_consensus.cc:3060] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.705300 2372 log.cc:1199] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/63a63b796191465aab9a63d3ffb0554d
I20251026 02:56:36.705839 2372 ts_tablet_manager.cc:1950] T 63a63b796191465aab9a63d3ffb0554d P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:36.706748 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 527) took 4518 ms. Trace:
I20251026 02:56:36.706898 2372 rpcz_store.cc:276] 1026 02:56:32.188611 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.188659 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:36.623972 (+4435313us) ts_tablet_manager.cc:1151] Deleting tablet 63a63b796191465aab9a63d3ffb0554d
1026 02:56:36.623979 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.667541 (+ 43562us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.698100 (+ 30559us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.706724 (+ 8624us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4435115}
I20251026 02:56:36.707505 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 63a63b796191465aab9a63d3ffb0554d (table default.Table_b389bf6a8e5d4ca39fb2c59f8103652c [id=d27916c5ae83463e868d14a5d22ae281]) successfully deleted
I20251026 02:56:36.707893 2372 tablet_replica.cc:333] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:36.708118 2372 raft_consensus.cc:2243] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:36.708519 2372 raft_consensus.cc:2272] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.709151 2372 ts_tablet_manager.cc:1916] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:36.712106 2375 raft_consensus.cc:359] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.712337 2375 raft_consensus.cc:385] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.712414 2375 raft_consensus.cc:740] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.712491 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 58d60bf2a9e546e6a9474a5af6d5273a
I20251026 02:56:36.712611 2375 consensus_queue.cc:260] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.713845 2375 ts_tablet_manager.cc:1434] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20251026 02:56:36.714103 2375 tablet_bootstrap.cc:492] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:36.715018 1663 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 5226d4516ce24ecda51dab35c86da132 (table default.Table_b32c2e53d837476d9a0d5ae8ea1843d0 [id=c8435cef755d4de69c33f599df1c0f75]). Expected version 4 got 3
I20251026 02:56:36.718581 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet ded06ba1f80c4433ae78687e5990edce from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:36.719206 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 244c03a9293242a48077bdaf90a47865
I20251026 02:56:36.721721 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 58d60bf2a9e546e6a9474a5af6d5273a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.721930 2089 tablet_service.cc:1552] Processing DeleteTablet for tablet 63a63b796191465aab9a63d3ffb0554d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.722084 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 244c03a9293242a48077bdaf90a47865 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.727264 2372 ts_tablet_manager.cc:1929] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:36.727320 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_34405bccc46d4867acc88741694d9255"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:36.727892 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34405bccc46d4867acc88741694d9255 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.729269 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9973c4eeef944fcb801d4e3d0013d538"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:36.729698 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9973c4eeef944fcb801d4e3d0013d538 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.731076 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_914472564539427caff89590b14d9d01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:36.731689 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_914472564539427caff89590b14d9d01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.734630 2372 log.cc:1199] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3791cb18fe1f4937badb4859d00647f3
I20251026 02:56:36.735214 2372 ts_tablet_manager.cc:1950] T 3791cb18fe1f4937badb4859d00647f3 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:36.736088 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 535) took 4308 ms. Trace:
I20251026 02:56:36.736254 2372 rpcz_store.cc:276] 1026 02:56:32.427691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.428181 (+ 490us) service_pool.cc:225] Handling call
1026 02:56:36.707842 (+4279661us) ts_tablet_manager.cc:1151] Deleting tablet 3791cb18fe1f4937badb4859d00647f3
1026 02:56:36.707853 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.712827 (+ 4974us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.727150 (+ 14323us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.736065 (+ 8915us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4279389}
I20251026 02:56:36.736963 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3791cb18fe1f4937badb4859d00647f3 (table default.Table_9fb0008346da44318ccc90cd69474750 [id=c7a6636ca94a4863a89fbef09c339bc1]) successfully deleted
I20251026 02:56:36.737406 2372 tablet_replica.cc:333] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:36.737639 2372 raft_consensus.cc:2243] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:36.737818 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc34ad9deb6d4fcb87d0c0fdfee17bbf consensus state has no leader
I20251026 02:56:36.737859 2372 pending_rounds.cc:70] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48: Trying to abort 1 pending ops.
I20251026 02:56:36.737926 2372 pending_rounds.cc:77] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888523555131392 op_type: NO_OP noop_request { }
I20251026 02:56:36.738014 2372 raft_consensus.cc:2889] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:36.738099 2372 raft_consensus.cc:2272] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.738236 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5a7f5adeea764cda87c60443f2e29e50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:36.738642 2372 ts_tablet_manager.cc:1916] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:36.739741 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a7f5adeea764cda87c60443f2e29e50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.744153 1663 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet ff242ef9541a42d793c4484190956da6 (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]). Expected version 1 got 0
I20251026 02:56:36.745138 2372 ts_tablet_manager.cc:1929] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:36.745337 2372 log.cc:1199] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/a08efd37968846d2958e54831488235c
I20251026 02:56:36.745826 2372 ts_tablet_manager.cc:1950] T a08efd37968846d2958e54831488235c P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:36.746703 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 539) took 4216 ms. Trace:
I20251026 02:56:36.746868 2372 rpcz_store.cc:276] 1026 02:56:32.529864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.529912 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:36.737338 (+4207426us) ts_tablet_manager.cc:1151] Deleting tablet a08efd37968846d2958e54831488235c
1026 02:56:36.737350 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.742056 (+ 4706us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.745028 (+ 2972us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.746679 (+ 1651us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4207177}
I20251026 02:56:36.747803 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet a08efd37968846d2958e54831488235c (table default.Table_80ca368a0ade4ac2b090b31b2ab56f12 [id=49dd7ccfa92946bebeaa4c178138f75a]) successfully deleted
I20251026 02:56:36.748191 2372 tablet_replica.cc:333] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:36.748477 2372 raft_consensus.cc:2243] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:36.749418 2372 pending_rounds.cc:70] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48: Trying to abort 1 pending ops.
I20251026 02:56:36.749517 2372 pending_rounds.cc:77] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888529224867840 op_type: NO_OP noop_request { }
I20251026 02:56:36.749656 2372 raft_consensus.cc:2889] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:36.749779 2372 raft_consensus.cc:2272] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.750830 2372 ts_tablet_manager.cc:1916] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:36.751725 3420 log.cc:927] Time spent T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20251026 02:56:36.752445 3401 tablet.cc:1722] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251026 02:56:36.754254 2375 tablet_bootstrap.cc:654] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.695927 3318 raft_consensus.cc:493] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.726097 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 03da415f3a944a9a9679b3a45295ce35 with delete_type TABLET_DATA_DELETED (replaced by tablet a6f9ae327aa4401baf66a737eaa93cf6) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:36.726248 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet ced7ad6a601d432cbc13b3199a0506e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:36.755546 3318 raft_consensus.cc:515] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.756299 3318 leader_election.cc:290] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:36.756636 2376 log.cc:1199] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/86d8e1e21c5f440d8baf76c357d01ad2
I20251026 02:56:36.756987 2376 ts_tablet_manager.cc:1950] T 86d8e1e21c5f440d8baf76c357d01ad2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:36.758255 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94886ad6d6bb45f39261725169774ac9" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:36.726559 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 122de8616b8f4fcf88971d5738c47e82 with delete_type TABLET_DATA_DELETED (replaced by tablet 81775055d98349f696cdab6171ffd58b) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:36.734508 2378 tablet_bootstrap.cc:492] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:36.761256 2378 ts_tablet_manager.cc:1403] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.000s
I20251026 02:56:36.762017 2378 raft_consensus.cc:359] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.762228 2378 raft_consensus.cc:385] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.762298 2378 raft_consensus.cc:740] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.762506 2378 consensus_queue.cc:260] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.762967 2378 ts_tablet_manager.cc:1434] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:36.763250 2378 tablet_bootstrap.cc:492] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:36.771916 2378 tablet_bootstrap.cc:654] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.774572 1980 raft_consensus.cc:3060] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.775413 3394 tablet.cc:1722] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251026 02:56:36.777822 2372 ts_tablet_manager.cc:1929] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:36.777971 2372 log.cc:1199] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/0fe0e06a86e241bc91a1db4219fe25bc
I20251026 02:56:36.739130 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 244c03a9293242a48077bdaf90a47865 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:36.744385 1947 tablet_service.cc:1552] Processing DeleteTablet for tablet 58d60bf2a9e546e6a9474a5af6d5273a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:36.794353 2377 ts_tablet_manager.cc:1929] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:36.798782 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 512) took 4161 ms. Trace:
I20251026 02:56:36.799311 2376 rpcz_store.cc:276] 1026 02:56:32.636971 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.637591 (+ 620us) service_pool.cc:225] Handling call
1026 02:56:36.548810 (+3911219us) ts_tablet_manager.cc:1151] Deleting tablet 86d8e1e21c5f440d8baf76c357d01ad2
1026 02:56:36.548820 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.626190 (+ 77370us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.689622 (+ 63432us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.798759 (+109137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40909,"tablet-delete.queue_time_us":3911034}
I20251026 02:56:36.800011 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 86d8e1e21c5f440d8baf76c357d01ad2 (table default.Table_689f802c9e4b457eb71fb78898fe14b0 [id=05242557de9a4a19948b8591ce73ab28]) successfully deleted
I20251026 02:56:36.800491 2376 tablet_replica.cc:333] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:36.800734 2376 raft_consensus.cc:2243] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:36.800940 2376 raft_consensus.cc:2272] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.801597 2376 ts_tablet_manager.cc:1916] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:36.794742 2257 raft_consensus.cc:2468] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:36.796501 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7062c5ff91e14369ab69a2d199585c55" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:36.806080 2259 raft_consensus.cc:3060] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:36.803445 2381 tablet_bootstrap.cc:492] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:36.807013 2378 tablet_bootstrap.cc:492] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:36.807145 2378 ts_tablet_manager.cc:1403] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.000s
W20251026 02:56:36.807785 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31edacb952514c8aa25c79f509869e8f consensus state has no leader
I20251026 02:56:36.811460 1659 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 94886ad6d6bb45f39261725169774ac9 (table default.Table_73789469dbaa4885b8566a3aa0877893 [id=9e897d33e6374ad4bc3c8acaae512645]). Expected version 2 got 0
I20251026 02:56:36.813263 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e0971fd722424b9cb2a4067114d03cfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:36.813612 2372 ts_tablet_manager.cc:1950] T 0fe0e06a86e241bc91a1db4219fe25bc P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:36.813661 2375 tablet_bootstrap.cc:492] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:36.813772 2375 ts_tablet_manager.cc:1403] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.000s
W20251026 02:56:36.814033 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0971fd722424b9cb2a4067114d03cfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.814469 2375 raft_consensus.cc:359] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.814623 2375 raft_consensus.cc:385] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.814663 2375 raft_consensus.cc:740] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.814836 2375 consensus_queue.cc:260] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.807885 2378 raft_consensus.cc:359] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.815140 2378 raft_consensus.cc:385] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.815215 2378 raft_consensus.cc:740] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.816416 2378 consensus_queue.cc:260] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.820629 3394 tablet.cc:1722] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:36.824463 2381 ts_tablet_manager.cc:1403] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.328s user 0.009s sys 0.002s
I20251026 02:56:36.807288 2377 log.cc:1199] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/22e4a433801a4cb58062d3e76c760b69
I20251026 02:56:36.834064 2378 ts_tablet_manager.cc:1434] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20251026 02:56:36.840548 2195 leader_election.cc:304] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.840619 2962 consensus_queue.cc:1048] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:36.840977 2195 leader_election.cc:304] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.841389 2195 leader_election.cc:304] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:36.841446 3366 raft_consensus.cc:2804] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 02:56:36.841747 2195 leader_election.cc:343] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ced7ad6a601d432cbc13b3199a0506e4
W20251026 02:56:36.842078 2195 leader_election.cc:343] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ced7ad6a601d432cbc13b3199a0506e4
I20251026 02:56:36.814404 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 538) took 4284 ms. Trace:
I20251026 02:56:36.843737 2372 rpcz_store.cc:276] 1026 02:56:32.529763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.529980 (+ 217us) service_pool.cc:225] Handling call
1026 02:56:36.748137 (+4218157us) ts_tablet_manager.cc:1151] Deleting tablet 0fe0e06a86e241bc91a1db4219fe25bc
1026 02:56:36.748148 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.774459 (+ 26311us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.777703 (+ 3244us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.814381 (+ 36678us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4217949}
I20251026 02:56:36.844216 3328 raft_consensus.cc:2804] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:36.844267 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 0fe0e06a86e241bc91a1db4219fe25bc (table default.Table_caee34582bea4cd1970c4cc59078f9d1 [id=0494cd534eb24adc94c907471cdb346a]) successfully deleted
I20251026 02:56:36.844597 2372 tablet_replica.cc:333] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:36.844805 2372 raft_consensus.cc:2243] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:36.844977 3028 raft_consensus.cc:2804] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:36.845727 2378 tablet_bootstrap.cc:492] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:36.846880 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e0b929f603b4c25b5b971ee9db683e8 consensus state has no leader
I20251026 02:56:36.850278 3272 raft_consensus.cc:493] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.850463 3272 raft_consensus.cc:515] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.851260 3272 leader_election.cc:290] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:36.852494 2375 ts_tablet_manager.cc:1434] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20251026 02:56:36.854091 2377 ts_tablet_manager.cc:1950] T 22e4a433801a4cb58062d3e76c760b69 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:36.854511 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 497) took 5724 ms. Trace:
I20251026 02:56:36.854621 2377 rpcz_store.cc:276] 1026 02:56:31.130152 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.132030 (+ 1878us) service_pool.cc:225] Handling call
1026 02:56:36.483517 (+5351487us) ts_tablet_manager.cc:1151] Deleting tablet 22e4a433801a4cb58062d3e76c760b69
1026 02:56:36.483543 (+ 26us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.659914 (+176371us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.794185 (+134271us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.854489 (+ 60304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5336915}
I20251026 02:56:36.854987 2377 tablet_replica.cc:333] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:36.855196 2377 raft_consensus.cc:2243] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:36.855610 2377 raft_consensus.cc:2272] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.857812 2381 raft_consensus.cc:359] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.858053 2381 raft_consensus.cc:385] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.858119 2381 raft_consensus.cc:740] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.858350 2375 tablet_bootstrap.cc:492] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:36.859704 3401 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:36.858343 2381 consensus_queue.cc:260] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.863834 3028 raft_consensus.cc:697] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:36.864327 2381 ts_tablet_manager.cc:1434] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20251026 02:56:36.864192 3028 consensus_queue.cc:237] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.864607 2381 tablet_bootstrap.cc:492] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:36.865020 3366 raft_consensus.cc:697] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:36.865762 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3327d26a2c2c4d0b86d98151f5940bd7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:36.865262 3366 consensus_queue.cc:237] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:36.870635 2053 leader_election.cc:343] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.872056 2375 tablet_bootstrap.cc:654] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.875226 2381 tablet_bootstrap.cc:654] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.878343 2259 raft_consensus.cc:2468] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:36.883155 2377 ts_tablet_manager.cc:1916] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:36.892182 2378 tablet_bootstrap.cc:654] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.902592 2375 tablet_bootstrap.cc:492] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:36.902870 2375 ts_tablet_manager.cc:1403] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20251026 02:56:36.904052 2378 tablet_bootstrap.cc:492] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:36.909248 2378 ts_tablet_manager.cc:1403] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.004s
I20251026 02:56:36.910261 3371 raft_consensus.cc:493] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.910424 3371 raft_consensus.cc:515] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.911028 3371 leader_election.cc:290] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:36.912086 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8862cd48a5744c99b32f365cb77fa7c4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:36.916065 2053 leader_election.cc:343] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.917099 2378 raft_consensus.cc:359] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.910251 2375 raft_consensus.cc:359] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.918689 2375 raft_consensus.cc:385] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.918897 2378 raft_consensus.cc:385] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.919073 2372 raft_consensus.cc:2272] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.908638 2376 ts_tablet_manager.cc:1929] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:36.920203 2372 ts_tablet_manager.cc:1916] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:36.924402 2376 log.cc:1199] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/dbfe80d388644d0488cb2431a334c908
I20251026 02:56:36.924921 2376 ts_tablet_manager.cc:1950] T dbfe80d388644d0488cb2431a334c908 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:36.925326 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 517) took 4048 ms. Trace:
I20251026 02:56:36.925619 2378 raft_consensus.cc:740] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.926215 2381 tablet_bootstrap.cc:492] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:36.925719 2376 rpcz_store.cc:276] 1026 02:56:32.877309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.879167 (+ 1858us) service_pool.cc:225] Handling call
1026 02:56:36.800438 (+3921271us) ts_tablet_manager.cc:1151] Deleting tablet dbfe80d388644d0488cb2431a334c908
1026 02:56:36.800450 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.867457 (+ 67007us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.908494 (+ 41037us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.925305 (+ 16811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3913524}
I20251026 02:56:36.926380 2381 ts_tablet_manager.cc:1403] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.000s
I20251026 02:56:36.927204 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet dbfe80d388644d0488cb2431a334c908 (table default.Table_ce7dea3ace314a5fa3ffeb9f16512725 [id=5964dceedb1c4d69b39051f9fe86b9dd]) successfully deleted
I20251026 02:56:36.927361 2381 raft_consensus.cc:359] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.927537 2381 raft_consensus.cc:385] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:36.927587 2381 raft_consensus.cc:740] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.927790 2381 consensus_queue.cc:260] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.926664 2378 consensus_queue.cc:260] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.928412 2381 ts_tablet_manager.cc:1434] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:36.928707 2381 tablet_bootstrap.cc:492] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:36.930398 2381 tablet_bootstrap.cc:654] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.928411 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 519) took 4033 ms. Trace:
I20251026 02:56:36.936774 2376 rpcz_store.cc:276] 1026 02:56:32.895034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.895125 (+ 91us) service_pool.cc:225] Handling call
1026 02:56:36.927781 (+4032656us) ts_tablet_manager.cc:1151] Deleting tablet 262f60fdb274476abf3533b4f47688b8
1026 02:56:36.927792 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.927904 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4032073}
W20251026 02:56:36.942649 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 262f60fdb274476abf3533b4f47688b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 262f60fdb274476abf3533b4f47688b8 already in progress: creating tablet
I20251026 02:56:36.943312 2376 tablet_replica.cc:333] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:36.948153 2376 raft_consensus.cc:2243] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:36.949151 2376 pending_rounds.cc:70] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: Trying to abort 1 pending ops.
I20251026 02:56:36.950287 2376 pending_rounds.cc:77] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888526705414144 op_type: NO_OP noop_request { }
I20251026 02:56:36.950660 2376 raft_consensus.cc:2889] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:36.950961 2376 raft_consensus.cc:2272] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:36.951020 1916 consensus_peers.cc:597] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e62064fbdffb4954a3a1227efd6327ee. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:36.942001 2378 ts_tablet_manager.cc:1434] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20251026 02:56:36.953512 3371 raft_consensus.cc:493] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.953720 3371 raft_consensus.cc:515] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.954713 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d00f935fbc4b4d80820018fa2857a22d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:36.955417 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d00f935fbc4b4d80820018fa2857a22d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.957710 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_18ed9c54e04542528f377fd10c75cf9c" } new_table_name: "default.Table_fb1f96ba0dc7492c97a2c1fe243e22af" modify_external_catalogs: true
I20251026 02:56:36.957856 3371 leader_election.cc:290] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:36.958546 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882cd1680aa140f6a63a961cd4aec25a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:36.959126 2053 leader_election.cc:343] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.959065 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b083b1c076f1492a93b6e8f74e8bc574"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:36.959611 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b083b1c076f1492a93b6e8f74e8bc574 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.962621 2378 tablet_bootstrap.cc:492] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:36.962666 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a79373c41ad7448e903a6d939c57b345" } new_table_name: "default.Table_ce57d8a514194198ad913d3eb35269b9" modify_external_catalogs: true
I20251026 02:56:36.963694 3371 raft_consensus.cc:493] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:36.963893 3371 raft_consensus.cc:515] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:36.964540 3371 leader_election.cc:290] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:36.965627 2377 ts_tablet_manager.cc:1929] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:36.965644 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be1c0948c0ae4563b7cb644521914cb9" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:36.966619 2053 leader_election.cc:343] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:36.968056 2376 ts_tablet_manager.cc:1916] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:36.919112 2375 raft_consensus.cc:740] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:36.969969 2375 consensus_queue.cc:260] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:36.970754 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4ded23ba48e4400b9b96e1dd82638b49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:36.971314 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ded23ba48e4400b9b96e1dd82638b49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:36.972258 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ded06ba1f80c4433ae78687e5990edce
I20251026 02:56:36.972787 1660 catalog_manager.cc:6719] ReplaceTablet: tablet ded06ba1f80c4433ae78687e5990edce deleted and replaced by tablet 3131bc2948e34c25a1bbda90f8dabeb3
I20251026 02:56:36.973675 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet 262f60fdb274476abf3533b4f47688b8 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:36.974375 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet ded06ba1f80c4433ae78687e5990edce with delete_type TABLET_DATA_DELETED (replaced by tablet 3131bc2948e34c25a1bbda90f8dabeb3) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.975553 1663 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_b32c2e53d837476d9a0d5ae8ea1843d0 [id=c8435cef755d4de69c33f599df1c0f75] alter complete (version 4)
W20251026 02:56:36.976447 1663 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet ff242ef9541a42d793c4484190956da6 (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ff242ef9541a42d793c4484190956da6 consensus state has no leader
I20251026 02:56:36.979643 2089 tablet_service.cc:1552] Processing DeleteTablet for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.979897 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 0fe0e06a86e241bc91a1db4219fe25bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.979974 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 7801f073d5d94076af9299492f86fad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:36.981781 2377 log.cc:1199] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/a111848706b743c9beb2fbd21f59d7f3
I20251026 02:56:36.982240 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet e830f1aa1270420c9b28a0491a1a0960 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:36.982297 2377 ts_tablet_manager.cc:1950] T a111848706b743c9beb2fbd21f59d7f3 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:36.983403 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_96ed23975f9842f9a636936ad4f2727d" } modify_external_catalogs: true
I20251026 02:56:36.983671 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_96ed23975f9842f9a636936ad4f2727d" } modify_external_catalogs: true
I20251026 02:56:36.985196 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_719b11bde46041c5a20750db14381807" } modify_external_catalogs: true
I20251026 02:56:36.985394 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_719b11bde46041c5a20750db14381807" } modify_external_catalogs: true
I20251026 02:56:36.986231 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7921be016a944a41953b751ba927dacd" } modify_external_catalogs: true
I20251026 02:56:36.986385 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7921be016a944a41953b751ba927dacd" } modify_external_catalogs: true
I20251026 02:56:36.988935 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 502) took 5805 ms. Trace:
I20251026 02:56:36.990067 2375 ts_tablet_manager.cc:1434] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.081s user 0.002s sys 0.000s
I20251026 02:56:36.990513 2375 tablet_bootstrap.cc:492] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:36.992823 2375 tablet_bootstrap.cc:654] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:36.989023 2377 rpcz_store.cc:276] 1026 02:56:31.183194 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.183520 (+ 326us) service_pool.cc:225] Handling call
1026 02:56:36.854942 (+5671422us) ts_tablet_manager.cc:1151] Deleting tablet a111848706b743c9beb2fbd21f59d7f3
1026 02:56:36.854950 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.936426 (+ 81476us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.965457 (+ 29031us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:36.988913 (+ 23456us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5671185}
I20251026 02:56:36.995687 2377 tablet_replica.cc:333] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:36.995945 2377 raft_consensus.cc:2243] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:36.996475 2377 pending_rounds.cc:70] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:36.996585 2377 pending_rounds.cc:77] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888527627177984 op_type: NO_OP noop_request { }
I20251026 02:56:36.996729 2377 raft_consensus.cc:2889] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:36.996893 2377 raft_consensus.cc:2272] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:36.997499 2377 ts_tablet_manager.cc:1916] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:37.002590 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64faa109f006414bb955ed483ecc189e consensus state has no leader
I20251026 02:56:37.003522 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_833e24776bb6450a92a7b7db2e1b2481" } modify_external_catalogs: true
I20251026 02:56:37.003705 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_833e24776bb6450a92a7b7db2e1b2481" } modify_external_catalogs: true
I20251026 02:56:37.004562 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fa3b58cf82f242cebd20cc1ea9b2133d" } modify_external_catalogs: true
I20251026 02:56:37.004719 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fa3b58cf82f242cebd20cc1ea9b2133d" } modify_external_catalogs: true
I20251026 02:56:37.013397 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 47037db3602246a2ab6224557286162f from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:37.013839 2372 ts_tablet_manager.cc:1929] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251026 02:56:37.013887 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3131bc2948e34c25a1bbda90f8dabeb3 has no consensus state
I20251026 02:56:36.974175 3318 raft_consensus.cc:493] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.015174 3318 raft_consensus.cc:515] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.015460 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2d0444a3af524d67ac524f71f4d70f11" } new_table_name: "default.Table_a923651b1a2746818e0bbbbaf1826bb3" modify_external_catalogs: true
I20251026 02:56:37.016356 1947 tablet_service.cc:1552] Processing DeleteTablet for tablet ded06ba1f80c4433ae78687e5990edce with delete_type TABLET_DATA_DELETED (replaced by tablet 3131bc2948e34c25a1bbda90f8dabeb3) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.016607 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet f6d6f024d8b943d5acf69b64bc7c7214 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a3491b303f94fe2a2c838297f73d464) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.017323 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_55f34abf69314dfab04ededf15472ee8" } new_table_name: "default.Table_28fda9a17d1a4e82bf90e8a71a271db8" modify_external_catalogs: true
I20251026 02:56:37.013975 2372 log.cc:1199] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/634d913c98c842a49d97a54f23752d26
I20251026 02:56:37.018640 2372 ts_tablet_manager.cc:1950] T 634d913c98c842a49d97a54f23752d26 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:37.019536 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 536) took 4490 ms. Trace:
I20251026 02:56:37.020174 2372 rpcz_store.cc:276] 1026 02:56:32.529491 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.530089 (+ 598us) service_pool.cc:225] Handling call
1026 02:56:36.844543 (+4314454us) ts_tablet_manager.cc:1151] Deleting tablet 634d913c98c842a49d97a54f23752d26
1026 02:56:36.844555 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:36.923820 (+ 79265us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.013697 (+ 89877us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.019489 (+ 5792us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74009,"tablet-delete.queue_time_us":4314108}
I20251026 02:56:37.020798 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_79f94fa4977d4066907d585cd301d523"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.021323 2378 tablet_bootstrap.cc:654] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:37.021471 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79f94fa4977d4066907d585cd301d523 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.022553 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 634d913c98c842a49d97a54f23752d26 (table default.Table_d2dcec579c684b92a92d98fb59ee3da1 [id=4f7edfb6d75c4a0b94ca99bb9cfe14f3]) successfully deleted
I20251026 02:56:37.023084 2372 tablet_replica.cc:333] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:37.023701 2372 raft_consensus.cc:2243] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:37.025209 2372 raft_consensus.cc:2272] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:37.026142 2372 ts_tablet_manager.cc:1916] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:37.026580 3318 leader_election.cc:290] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.030396 2375 tablet_bootstrap.cc:492] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:37.030630 2375 ts_tablet_manager.cc:1403] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
W20251026 02:56:37.031533 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e3cdb1f81214337a7ccc7816755f58c consensus state has no leader
I20251026 02:56:37.031612 2375 raft_consensus.cc:359] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.031934 2375 raft_consensus.cc:385] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.032052 2375 raft_consensus.cc:740] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.032459 2375 consensus_queue.cc:260] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:37.033382 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91f81d156ff44decb63a4c0f5eacdac9 consensus state has no leader
I20251026 02:56:37.033679 2375 ts_tablet_manager.cc:1434] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:37.034047 2375 tablet_bootstrap.cc:492] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:37.036073 2375 tablet_bootstrap.cc:654] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:37.036828 1916 leader_election.cc:343] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ca4611c0a494425ca50d599b310872c8
I20251026 02:56:37.048432 1657 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_26dd741c83ff419f8c9bc808e83d9c72 [id=c162ef89d2f14e91a24a6d8c2c37c261] alter complete (version 5)
I20251026 02:56:37.057682 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d9351d4a858248d8813e9df05f900010" } new_table_name: "default.Table_f65b2167c56c419b9b5fd835ce39e3ed" modify_external_catalogs: true
I20251026 02:56:37.059468 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_10236dd7e7264f888b32afcb8c56c95f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.060019 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10236dd7e7264f888b32afcb8c56c95f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.060804 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e56b259a314422aba35eb129021bbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.061007 1947 tablet_service.cc:1552] Processing DeleteTablet for tablet f85c2745a371455a8dda67f31a19c9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.061568 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_63579add544846d3b2a9f267e24fda8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.062167 2377 ts_tablet_manager.cc:1929] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:37.062294 2377 log.cc:1199] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/276bd3b97c97426daf802e1f57feb265
W20251026 02:56:37.062389 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63579add544846d3b2a9f267e24fda8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:37.072459 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d01fa8b270e1476cb86d81a196ba94f8 consensus state has no leader
I20251026 02:56:37.073019 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6b7d3c3d1bbc40f9b1ef1a34f2f5332f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.073262 1662 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 882cd1680aa140f6a63a961cd4aec25a (table default.Table_52417f25dbdb4676a5aaaca9bf46962c [id=cc7a8c37a3b342d988872e3baa16a26d]). Expected version 1 got 0
I20251026 02:56:37.073477 1662 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 5c6e3e47b3e7421e945cfbe671b06abe (table default.Table_653a3f85c24140cdb62a3e9f3ea95fd4 [id=a305da1e3a694b32aff05dbd6c58a6cb]). Expected version 1 got 0
W20251026 02:56:37.073658 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b7d3c3d1bbc40f9b1ef1a34f2f5332f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.074391 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet 1ffead7ca2d946c78aa4b8a602865fc8 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:37.085850 1662 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 5c6e3e47b3e7421e945cfbe671b06abe (table default.Table_653a3f85c24140cdb62a3e9f3ea95fd4 [id=a305da1e3a694b32aff05dbd6c58a6cb]). Expected version 1 got 0
I20251026 02:56:37.086489 2095 tablet_service.cc:1505] Processing CreateTablet for tablet b68388b2fd014dd1a5f6572796cf53a6 (DEFAULT_TABLE table=default.Table_833e24776bb6450a92a7b7db2e1b2481 [id=7065537e6fe649c5b53c099ad6aa413d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.087046 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b68388b2fd014dd1a5f6572796cf53a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.087754 2096 tablet_service.cc:1505] Processing CreateTablet for tablet c6a263a68d6e4b5d8d35c78595266e2b (DEFAULT_TABLE table=default.Table_7a6f36e22379480f957ea030136b008f [id=af8459b689814d1d824b503316ea179b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.088191 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a263a68d6e4b5d8d35c78595266e2b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:37.063067 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 220681us
I20251026 02:56:37.083847 2381 tablet_bootstrap.cc:492] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:37.096472 2381 ts_tablet_manager.cc:1403] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.168s user 0.007s sys 0.015s
I20251026 02:56:37.097590 2381 raft_consensus.cc:359] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.097889 2381 raft_consensus.cc:385] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.098022 2381 raft_consensus.cc:740] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.099182 1947 tablet_service.cc:1505] Processing CreateTablet for tablet b68388b2fd014dd1a5f6572796cf53a6 (DEFAULT_TABLE table=default.Table_833e24776bb6450a92a7b7db2e1b2481 [id=7065537e6fe649c5b53c099ad6aa413d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.099694 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b68388b2fd014dd1a5f6572796cf53a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.099835 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce922040298d4952af13652eeda7a234" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:37.099970 1915 leader_election.cc:304] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:37.100064 2120 raft_consensus.cc:2468] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:37.100261 1954 tablet_service.cc:1505] Processing CreateTablet for tablet c6a263a68d6e4b5d8d35c78595266e2b (DEFAULT_TABLE table=default.Table_7a6f36e22379480f957ea030136b008f [id=af8459b689814d1d824b503316ea179b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.100462 3372 raft_consensus.cc:2749] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.101555 1915 leader_election.cc:304] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:37.101953 3372 raft_consensus.cc:2749] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.103644 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1a36b8a4beca483bb853d27961f5281a
I20251026 02:56:37.104907 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a36b8a4beca483bb853d27961f5281a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.105170 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a263a68d6e4b5d8d35c78595266e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.106145 2053 leader_election.cc:304] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.108870 3272 raft_consensus.cc:2804] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:37.098311 2381 consensus_queue.cc:260] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.109076 3272 raft_consensus.cc:697] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:37.109488 2381 ts_tablet_manager.cc:1434] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251026 02:56:37.109402 3272 consensus_queue.cc:237] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.109850 2381 tablet_bootstrap.cc:492] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:37.112277 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 22e4a433801a4cb58062d3e76c760b69 (table default.Table_f7c216f3ebd34c718ceef042a435b4cd [id=b4c0611288a949378eed16d32b5170b9]) successfully deleted
I20251026 02:56:37.113171 2381 tablet_bootstrap.cc:654] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.084266 3328 raft_consensus.cc:697] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:37.114969 3328 consensus_queue.cc:237] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.119640 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7e1a45412db1449a99252112d8110a17
I20251026 02:56:37.120527 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e1a45412db1449a99252112d8110a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.125469 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a36b8a4beca483bb853d27961f5281a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.126863 2053 leader_election.cc:304] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.128547 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b68388b2fd014dd1a5f6572796cf53a6
I20251026 02:56:37.129331 3371 raft_consensus.cc:2804] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:37.129532 3371 raft_consensus.cc:697] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:37.129897 3371 consensus_queue.cc:237] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.132414 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet b68388b2fd014dd1a5f6572796cf53a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.132452 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet b68388b2fd014dd1a5f6572796cf53a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.134634 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 81775055d98349f696cdab6171ffd58b
I20251026 02:56:37.135433 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 81775055d98349f696cdab6171ffd58b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.137955 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet 81775055d98349f696cdab6171ffd58b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.138648 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e1a45412db1449a99252112d8110a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.146332 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet a111848706b743c9beb2fbd21f59d7f3 (table default.Table_1b3d02c229434ae4a351fd493a20654b [id=2d5f1153b96d45cbbe2e74da79045dbf]) successfully deleted
I20251026 02:56:37.147897 2378 tablet_bootstrap.cc:492] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:37.148147 2378 ts_tablet_manager.cc:1403] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.186s user 0.001s sys 0.006s
I20251026 02:56:37.149379 2378 raft_consensus.cc:359] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.149718 2378 raft_consensus.cc:385] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.149842 2378 raft_consensus.cc:740] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.150240 2378 consensus_queue.cc:260] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.151847 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet e32a1f8dcad4430eb35d388bfad7d110
I20251026 02:56:37.152444 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet e830f1aa1270420c9b28a0491a1a0960
I20251026 02:56:37.152909 1661 catalog_manager.cc:6719] ReplaceTablet: tablet e830f1aa1270420c9b28a0491a1a0960 deleted and replaced by tablet 8423becb5b214f32a66445cd351b200f
I20251026 02:56:37.157351 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet e32a1f8dcad4430eb35d388bfad7d110 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.157886 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet e830f1aa1270420c9b28a0491a1a0960 with delete_type TABLET_DATA_DELETED (replaced by tablet 8423becb5b214f32a66445cd351b200f) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.158229 1947 tablet_service.cc:1552] Processing DeleteTablet for tablet e830f1aa1270420c9b28a0491a1a0960 with delete_type TABLET_DATA_DELETED (replaced by tablet 8423becb5b214f32a66445cd351b200f) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.157807 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet e32a1f8dcad4430eb35d388bfad7d110 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.160424 2378 ts_tablet_manager.cc:1434] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251026 02:56:37.160456 2381 tablet_bootstrap.cc:492] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:37.160610 2381 ts_tablet_manager.cc:1403] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.003s
I20251026 02:56:37.161461 2381 raft_consensus.cc:359] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.161662 2381 raft_consensus.cc:385] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.161726 2381 raft_consensus.cc:740] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.162007 2381 consensus_queue.cc:260] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.162154 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 634d913c98c842a49d97a54f23752d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.162886 2377 ts_tablet_manager.cc:1950] T 276bd3b97c97426daf802e1f57feb265 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:37.163247 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 504) took 5857 ms. Trace:
I20251026 02:56:37.163343 2377 rpcz_store.cc:276] 1026 02:56:31.305960 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.306014 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:36.995633 (+5689619us) ts_tablet_manager.cc:1151] Deleting tablet 276bd3b97c97426daf802e1f57feb265
1026 02:56:36.995644 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.058307 (+ 62663us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.062063 (+ 3756us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.163231 (+101168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5689371}
I20251026 02:56:37.163904 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 505) took 5798 ms. Trace:
I20251026 02:56:37.164001 2377 rpcz_store.cc:276] 1026 02:56:31.365640 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.365714 (+ 74us) service_pool.cc:225] Handling call
1026 02:56:37.163756 (+5798042us) ts_tablet_manager.cc:1151] Deleting tablet 4b5e61afe90e4bc4a2897de21e0b3919
1026 02:56:37.163769 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.163885 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5797794}
I20251026 02:56:37.167411 1978 raft_consensus.cc:3060] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.170600 2378 tablet_bootstrap.cc:492] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:37.173108 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_09e15e381f1d41dca2fb4079688213db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.173923 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09e15e381f1d41dca2fb4079688213db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.176671 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_906d1b8fb76f4fd69325086bafba430d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.177278 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_906d1b8fb76f4fd69325086bafba430d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.184429 2381 ts_tablet_manager.cc:1434] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:37.185348 2381 tablet_bootstrap.cc:492] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:37.186918 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 54ab132fb4514e009ce6b32247291a0e (DEFAULT_TABLE table=default.Table_ff7d398f97d04b29873f9eacd1c9fb11 [id=ab33f4aecde5491a8224d1786f677517]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.187479 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ab132fb4514e009ce6b32247291a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.188521 2095 tablet_service.cc:1505] Processing CreateTablet for tablet e16fbc759b6b4b4ab7457000d1509419 (DEFAULT_TABLE table=default.Table_6f081e5839af41399767123dd4a4e5d8 [id=e285d129ee88497fb37b70bf16ca47ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.188922 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e16fbc759b6b4b4ab7457000d1509419. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.190339 3006 raft_consensus.cc:493] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.190531 3006 raft_consensus.cc:515] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.191208 3006 leader_election.cc:290] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.191774 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30c619ad37364747b6ebf97a73ef45a2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:37.192268 2053 leader_election.cc:343] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.194861 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet c13d89fc829049428363ede24c350ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.195127 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet f719c720cc474181a63786a3e248ab2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.195199 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 98fe44aab33b40eba819386c3bffca46 with delete_type TABLET_DATA_DELETED (replaced by tablet 03aed9babfbe4acdad21d8ee22dbac1d) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.195544 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 58d60bf2a9e546e6a9474a5af6d5273a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.195820 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 244c03a9293242a48077bdaf90a47865 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.196184 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet ded06ba1f80c4433ae78687e5990edce with delete_type TABLET_DATA_DELETED (replaced by tablet 3131bc2948e34c25a1bbda90f8dabeb3) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.196346 1980 raft_consensus.cc:3060] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.196928 2242 tablet_service.cc:1505] Processing CreateTablet for tablet b68388b2fd014dd1a5f6572796cf53a6 (DEFAULT_TABLE table=default.Table_833e24776bb6450a92a7b7db2e1b2481 [id=7065537e6fe649c5b53c099ad6aa413d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.197417 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b68388b2fd014dd1a5f6572796cf53a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.197510 3006 raft_consensus.cc:493] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.197675 3006 raft_consensus.cc:515] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.198922 3006 leader_election.cc:290] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.200990 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c6e3e47b3e7421e945cfbe671b06abe" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:37.202129 2381 tablet_bootstrap.cc:654] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:37.202210 2053 leader_election.cc:343] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.208709 3006 raft_consensus.cc:493] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.208968 3006 raft_consensus.cc:515] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.209898 3006 leader_election.cc:290] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.211617 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a345a2617d614092bedde12cb0020ce8" } new_table_name: "default.Table_fa22c9dc29814d9cb5c4bfdec072a3cf" modify_external_catalogs: true
I20251026 02:56:37.211658 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed836bb7519e4b0290e6547fa57128bc" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:37.211674 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6fc619ce52194bd58287fde601a7a24d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.212415 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fc619ce52194bd58287fde601a7a24d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:37.212780 2053 leader_election.cc:343] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.213006 3239 raft_consensus.cc:493] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.213150 3239 raft_consensus.cc:515] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.213232 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3788e42710a0457c80f6ae601866b2ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.213708 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3788e42710a0457c80f6ae601866b2ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.213778 3239 leader_election.cc:290] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:37.214375 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d2cc6b16d34331bb80c576c0cd2ce0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:37.214895 2194 leader_election.cc:343] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.216161 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_50ea87134e5c4bba934f9f7a5e275c9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.216692 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50ea87134e5c4bba934f9f7a5e275c9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.218134 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_455479b9f17040e48d4cd186bacef556" } new_table_name: "default.Table_a27cf0a8326c4683a305ae448369f2cd" modify_external_catalogs: true
I20251026 02:56:37.218652 2096 tablet_service.cc:1505] Processing CreateTablet for tablet cae444354939400f9b75440b26ccdd02 (DEFAULT_TABLE table=default.Table_720e8839bcb647ce8e030119c884cffa [id=c65d64b8fe6043c5b7bac9f00fc83695]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.219245 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cae444354939400f9b75440b26ccdd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.222816 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_26f3ddff250042de9c91eff90955f4ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.223505 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26f3ddff250042de9c91eff90955f4ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.224820 1947 tablet_service.cc:1505] Processing CreateTablet for tablet cae444354939400f9b75440b26ccdd02 (DEFAULT_TABLE table=default.Table_720e8839bcb647ce8e030119c884cffa [id=c65d64b8fe6043c5b7bac9f00fc83695]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.226253 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cae444354939400f9b75440b26ccdd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.228638 1954 tablet_service.cc:1505] Processing CreateTablet for tablet e16fbc759b6b4b4ab7457000d1509419 (DEFAULT_TABLE table=default.Table_6f081e5839af41399767123dd4a4e5d8 [id=e285d129ee88497fb37b70bf16ca47ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.229005 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e16fbc759b6b4b4ab7457000d1509419. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:37.231220 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6979455632e54d2081065954771f34d3 consensus state has no leader
I20251026 02:56:37.164403 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 506) took 5790 ms. Trace:
I20251026 02:56:37.235922 2377 rpcz_store.cc:276] 1026 02:56:31.374065 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.374345 (+ 280us) service_pool.cc:225] Handling call
1026 02:56:37.164265 (+5789920us) ts_tablet_manager.cc:1151] Deleting tablet 9a345fa5edbd48d2891b8b40b09cb3f8
1026 02:56:37.164275 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.164387 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5789680}
I20251026 02:56:37.236506 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 507) took 5860 ms. Trace:
I20251026 02:56:37.236653 2377 rpcz_store.cc:276] 1026 02:56:31.376384 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.376437 (+ 53us) service_pool.cc:225] Handling call
1026 02:56:37.236363 (+5859926us) ts_tablet_manager.cc:1151] Deleting tablet a3af2695c41a4439bc4092b7cb1b2edf
1026 02:56:37.236376 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.236488 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5859694}
I20251026 02:56:37.237063 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 508) took 5844 ms. Trace:
I20251026 02:56:37.237182 2377 rpcz_store.cc:276] 1026 02:56:31.392963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.393038 (+ 75us) service_pool.cc:225] Handling call
1026 02:56:37.236964 (+5843926us) ts_tablet_manager.cc:1151] Deleting tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5
1026 02:56:37.236972 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.237048 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5843669}
I20251026 02:56:37.237500 2377 tablet_replica.cc:333] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539: stopping tablet replica
W20251026 02:56:37.237571 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 consensus state has no leader
I20251026 02:56:37.237830 2377 raft_consensus.cc:2243] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:37.238090 2377 raft_consensus.cc:2272] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:37.239029 2377 ts_tablet_manager.cc:1916] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:37.243999 3006 raft_consensus.cc:493] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.244153 3006 raft_consensus.cc:515] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.244822 3006 leader_election.cc:290] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.245725 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:37.246327 2053 leader_election.cc:343] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.247911 3239 raft_consensus.cc:493] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.247864 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_71a2681a68ef4299afc9e7d30389b870" } new_table_name: "default.Table_c51b98a0c9e54daf9d1e457f34d56475" modify_external_catalogs: true
I20251026 02:56:37.248126 3239 raft_consensus.cc:515] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.249186 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:37.249262 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_28fda9a17d1a4e82bf90e8a71a271db8" } new_table_name: "default.Table_ac84ba7eb22a4275b759937b21a2a06c" modify_external_catalogs: true
W20251026 02:56:37.249776 2194 leader_election.cc:343] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.250161 3239 leader_election.cc:290] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:37.252605 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b67009abd92345148b7cb2d1be01e4a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.253193 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b67009abd92345148b7cb2d1be01e4a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.253469 3006 raft_consensus.cc:493] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.253612 3006 raft_consensus.cc:515] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.254232 3006 leader_election.cc:290] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.256750 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a378e913112c443db869016e3b98fd0e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:37.258996 2053 leader_election.cc:343] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.260068 1974 raft_consensus.cc:3060] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.265558 3006 raft_consensus.cc:493] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.265720 3006 raft_consensus.cc:515] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.266403 3006 leader_election.cc:290] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.267356 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c3050e72d42417ab58a05f7e3aa56b3" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:37.267951 2053 leader_election.cc:343] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:37.268860 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e3cdb1f81214337a7ccc7816755f58c consensus state has no leader
W20251026 02:56:37.269562 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8fe7825fa8d46669772cf84bc4b3afc consensus state has no leader
I20251026 02:56:37.271759 3372 raft_consensus.cc:493] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.271904 3372 raft_consensus.cc:515] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.272593 3372 leader_election.cc:290] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:37.273981 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 54ab132fb4514e009ce6b32247291a0e (DEFAULT_TABLE table=default.Table_ff7d398f97d04b29873f9eacd1c9fb11 [id=ab33f4aecde5491a8224d1786f677517]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.276537 2242 tablet_service.cc:1505] Processing CreateTablet for tablet c6a263a68d6e4b5d8d35c78595266e2b (DEFAULT_TABLE table=default.Table_7a6f36e22379480f957ea030136b008f [id=af8459b689814d1d824b503316ea179b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.279083 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a36b8a4beca483bb853d27961f5281a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.279371 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e1a45412db1449a99252112d8110a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.279569 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet b68388b2fd014dd1a5f6572796cf53a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.280329 2375 tablet_bootstrap.cc:492] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:37.282572 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ff6f3f84ac2348b79403d10fc9e3ecb5" } new_table_name: "default.Table_dc1b77e5238847adbd80a2df2274c003" modify_external_catalogs: true
W20251026 02:56:37.286070 1916 leader_election.cc:343] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: f85c2745a371455a8dda67f31a19c9d5
I20251026 02:56:37.282410 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 0cd431fcd27c4d3db2ea3dc4c265755d (DEFAULT_TABLE table=default.Table_e75a50dda2414622962b8832fdf3cb83 [id=518bbe9c3b674d74b088ff312ad6eb87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.287249 2375 ts_tablet_manager.cc:1403] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.253s user 0.002s sys 0.004s
I20251026 02:56:37.288854 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 81775055d98349f696cdab6171ffd58b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.289122 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet e830f1aa1270420c9b28a0491a1a0960 with delete_type TABLET_DATA_DELETED (replaced by tablet 8423becb5b214f32a66445cd351b200f) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.289392 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 0cd431fcd27c4d3db2ea3dc4c265755d (DEFAULT_TABLE table=default.Table_e75a50dda2414622962b8832fdf3cb83 [id=518bbe9c3b674d74b088ff312ad6eb87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.289537 2224 tablet_service.cc:1505] Processing CreateTablet for tablet cae444354939400f9b75440b26ccdd02 (DEFAULT_TABLE table=default.Table_720e8839bcb647ce8e030119c884cffa [id=c65d64b8fe6043c5b7bac9f00fc83695]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.289944 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cae444354939400f9b75440b26ccdd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.290409 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cd431fcd27c4d3db2ea3dc4c265755d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:37.290995 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3ac38e6257e74367b06b15fc49d2af81 consensus state has no leader
I20251026 02:56:37.292752 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet e32a1f8dcad4430eb35d388bfad7d110 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.292930 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 352a284902c94f088e2e6ae006162db0 (table default.Table_6cd4ee82e9ec406f971669b7f0722677 [id=414375003cd44b5c8c9f0249b339c28a]). Expected version 1 got 0
I20251026 02:56:37.293052 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 0cd431fcd27c4d3db2ea3dc4c265755d (DEFAULT_TABLE table=default.Table_e75a50dda2414622962b8832fdf3cb83 [id=518bbe9c3b674d74b088ff312ad6eb87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.293207 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 54ab132fb4514e009ce6b32247291a0e (DEFAULT_TABLE table=default.Table_ff7d398f97d04b29873f9eacd1c9fb11 [id=ab33f4aecde5491a8224d1786f677517]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.293411 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cd431fcd27c4d3db2ea3dc4c265755d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.293467 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 5226d4516ce24ecda51dab35c86da132 (table default.Table_b32c2e53d837476d9a0d5ae8ea1843d0 [id=c8435cef755d4de69c33f599df1c0f75]). Expected version 4 got 3
I20251026 02:56:37.293736 2243 tablet_service.cc:1505] Processing CreateTablet for tablet e16fbc759b6b4b4ab7457000d1509419 (DEFAULT_TABLE table=default.Table_6f081e5839af41399767123dd4a4e5d8 [id=e285d129ee88497fb37b70bf16ca47ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.294065 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e16fbc759b6b4b4ab7457000d1509419. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.297299 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ab132fb4514e009ce6b32247291a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.302199 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_111996853fae40db821e68942c9fae5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.302847 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_111996853fae40db821e68942c9fae5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.302199 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f25a397301e7496a8d70b9a34ae32f6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.304643 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f25a397301e7496a8d70b9a34ae32f6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.306607 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0198232bf3d14a08b9ef2331469c256c" } modify_external_catalogs: true
I20251026 02:56:37.306874 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0198232bf3d14a08b9ef2331469c256c" } modify_external_catalogs: true
I20251026 02:56:37.309703 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94886ad6d6bb45f39261725169774ac9" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.311775 1962 tablet_service.cc:1505] Processing CreateTablet for tablet ec94273925e94fb09f55249426bc4eaa (DEFAULT_TABLE table=default.Table_8248296e6b4a45d497271230d96ac519 [id=4caa9908fba349ef9ed1da38ac8fbe7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.312156 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec94273925e94fb09f55249426bc4eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.315245 2254 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f85c2745a371455a8dda67f31a19c9d5" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.315765 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8862cd48a5744c99b32f365cb77fa7c4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.316062 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882cd1680aa140f6a63a961cd4aec25a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.316365 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be1c0948c0ae4563b7cb644521914cb9" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.318053 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3327d26a2c2c4d0b86d98151f5940bd7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.318218 2259 raft_consensus.cc:2468] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:37.319500 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_517fcc0e025a4b8f9ec1a392735037a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.320065 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_517fcc0e025a4b8f9ec1a392735037a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.324376 3239 consensus_queue.cc:1048] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:37.325438 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f0cdd3958d9d4428954284d993884629" } modify_external_catalogs: true
I20251026 02:56:37.325651 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f0cdd3958d9d4428954284d993884629" } modify_external_catalogs: true
I20251026 02:56:37.288084 2375 raft_consensus.cc:359] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.327354 2375 raft_consensus.cc:385] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.328531 2375 raft_consensus.cc:740] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.333870 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fbd4d396350c48129da0be37e6c3bfc8" } modify_external_catalogs: true
I20251026 02:56:37.334110 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fbd4d396350c48129da0be37e6c3bfc8" } modify_external_catalogs: true
I20251026 02:56:37.336845 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5af1941b0082444c8bd348a080af20ee" } modify_external_catalogs: true
I20251026 02:56:37.337025 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5af1941b0082444c8bd348a080af20ee" } modify_external_catalogs: true
I20251026 02:56:37.338949 2375 consensus_queue.cc:260] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.339664 2375 ts_tablet_manager.cc:1434] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20251026 02:56:37.340018 2375 tablet_bootstrap.cc:492] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:37.342900 1647 consensus_peers.cc:597] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 -> Peer 751b719b75ce4bc2a88a19091230ecb7 (127.1.150.60:33729): Couldn't send request to peer 751b719b75ce4bc2a88a19091230ecb7. Status: Network error: Client connection negotiation failed: client connection to 127.1.150.60:33729: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251026 02:56:37.345393 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b8cad7adae7e43469ceadbedeee53564
I20251026 02:56:37.346493 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 0b8b0b75eb394e9d9a91f51156195912 (DEFAULT_TABLE table=default.Table_adc311a814324b2db4e8aee6caa91f09 [id=418104d67c844ac19ad891922155dd86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.348096 2089 tablet_service.cc:1505] Processing CreateTablet for tablet ec94273925e94fb09f55249426bc4eaa (DEFAULT_TABLE table=default.Table_8248296e6b4a45d497271230d96ac519 [id=4caa9908fba349ef9ed1da38ac8fbe7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.349334 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30c619ad37364747b6ebf97a73ef45a2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.349951 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 0b8b0b75eb394e9d9a91f51156195912 (DEFAULT_TABLE table=default.Table_adc311a814324b2db4e8aee6caa91f09 [id=418104d67c844ac19ad891922155dd86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.350400 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b8b0b75eb394e9d9a91f51156195912. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.351536 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c6e3e47b3e7421e945cfbe671b06abe" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.352166 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed836bb7519e4b0290e6547fa57128bc" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.353404 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.353575 2255 raft_consensus.cc:2468] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:37.353768 1656 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet c44e8b5c98884a36b2ac43bca87bf014
I20251026 02:56:37.354614 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a378e913112c443db869016e3b98fd0e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.354765 2255 raft_consensus.cc:2468] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:37.355291 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c3050e72d42417ab58a05f7e3aa56b3" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.358117 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet b8cad7adae7e43469ceadbedeee53564 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.362649 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ba08469339184b11b9ced9e6bdd63d42
I20251026 02:56:37.363152 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet b8cad7adae7e43469ceadbedeee53564 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.364127 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet ba08469339184b11b9ced9e6bdd63d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.364660 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet ba08469339184b11b9ced9e6bdd63d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.365736 3371 raft_consensus.cc:493] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.365938 3371 raft_consensus.cc:515] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.366806 3371 leader_election.cc:290] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.372042 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet e701d4023fee45d18c48997aab3227c2 (table default.Table_464e60af830f4416af8fdcf84f660a49 [id=de836edc225f41d8b0c68748936306ad]). Expected version 2 got 0
I20251026 02:56:37.373405 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 6108219254bd448c852096c0c967384b
I20251026 02:56:37.374662 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 6108219254bd448c852096c0c967384b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.375152 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 6108219254bd448c852096c0c967384b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.376256 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d450815506d645aab06dee257de7ea3e" } modify_external_catalogs: true
I20251026 02:56:37.376652 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d450815506d645aab06dee257de7ea3e" } modify_external_catalogs: true
I20251026 02:56:37.380390 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dbe8002df6c4b64b8ec8260ca6b9cf5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:37.381155 2053 leader_election.cc:343] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.384058 2378 tablet_bootstrap.cc:654] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.385915 2381 tablet_bootstrap.cc:492] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:37.386260 2381 ts_tablet_manager.cc:1403] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.201s user 0.004s sys 0.003s
I20251026 02:56:37.387254 2381 raft_consensus.cc:359] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.387521 2381 raft_consensus.cc:385] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.387634 2381 raft_consensus.cc:740] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.387926 2381 consensus_queue.cc:260] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.388631 2381 ts_tablet_manager.cc:1434] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:37.388973 2381 tablet_bootstrap.cc:492] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:37.390965 2381 tablet_bootstrap.cc:654] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.393622 3272 raft_consensus.cc:493] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.394160 3272 raft_consensus.cc:515] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.396823 3272 leader_election.cc:290] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.397691 3318 raft_consensus.cc:493] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.397830 3318 raft_consensus.cc:515] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.398439 3318 leader_election.cc:290] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:37.399783 1916 leader_election.cc:343] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 634d913c98c842a49d97a54f23752d26
I20251026 02:56:37.400394 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff242ef9541a42d793c4484190956da6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:37.400943 2053 leader_election.cc:343] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.401011 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 3e424a9c89e64b1c98ada17fa5c31e9e (DEFAULT_TABLE table=default.Table_b12aefa4eb5b4380a6b9a31290cd24dc [id=1a4bff2fe09e4c2588c2ede9c8d95757]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.403138 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 3e424a9c89e64b1c98ada17fa5c31e9e (DEFAULT_TABLE table=default.Table_b12aefa4eb5b4380a6b9a31290cd24dc [id=1a4bff2fe09e4c2588c2ede9c8d95757]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.403816 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e424a9c89e64b1c98ada17fa5c31e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.422086 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 276bd3b97c97426daf802e1f57feb265 (table default.Table_3babf04294cc48ef92df6ffe8ab43e5a [id=e91358e10bab4589a6f4e815a85a5f7d]) successfully deleted
I20251026 02:56:37.422657 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e74211bb29f145a8aeec9c6f0dab4bff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.332878 2372 ts_tablet_manager.cc:1929] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:37.424336 2372 log.cc:1199] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/863ce9e1adb444b7b1399b968ea79bb4
I20251026 02:56:37.424780 2372 ts_tablet_manager.cc:1950] T 863ce9e1adb444b7b1399b968ea79bb4 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:37.424607 1663 catalog_manager.cc:5649] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:37.425269 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a463861b69f34f099edf69c1eb7d99a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.425663 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 537) took 4896 ms. Trace:
I20251026 02:56:37.425763 2372 rpcz_store.cc:276] 1026 02:56:32.529661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.530481 (+ 820us) service_pool.cc:225] Handling call
1026 02:56:37.022991 (+4492510us) ts_tablet_manager.cc:1151] Deleting tablet 863ce9e1adb444b7b1399b968ea79bb4
1026 02:56:37.023007 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.072112 (+ 49105us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.332743 (+260631us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.425641 (+ 92898us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4492279}
I20251026 02:56:37.426316 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 863ce9e1adb444b7b1399b968ea79bb4 (table default.Table_03bc36a9fd184c28abd1caf6e5ec4f7a [id=ee13c495b56c48649ceacc8e1122efb4]) successfully deleted
W20251026 02:56:37.426479 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 4b5e61afe90e4bc4a2897de21e0b3919 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b5e61afe90e4bc4a2897de21e0b3919 already in progress: creating tablet
I20251026 02:56:37.427062 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7f5cd37c87f34a40bf05829af6de0473
I20251026 02:56:37.426733 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 548) took 4539 ms. Trace:
I20251026 02:56:37.427357 2372 rpcz_store.cc:276] 1026 02:56:32.887449 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.891045 (+ 3596us) service_pool.cc:225] Handling call
1026 02:56:37.426604 (+4535559us) ts_tablet_manager.cc:1151] Deleting tablet 262f60fdb274476abf3533b4f47688b8
1026 02:56:37.426615 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.426717 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4535318}
I20251026 02:56:37.429121 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f5cd37c87f34a40bf05829af6de0473 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:37.430846 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a463861b69f34f099edf69c1eb7d99a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:37.430799 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e74211bb29f145a8aeec9c6f0dab4bff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.430056 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f5cd37c87f34a40bf05829af6de0473 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.438400 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 549) took 4539 ms. Trace:
W20251026 02:56:37.439002 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 262f60fdb274476abf3533b4f47688b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 262f60fdb274476abf3533b4f47688b8 already in progress: creating tablet
I20251026 02:56:37.438607 2372 rpcz_store.cc:276] 1026 02:56:32.899085 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.899149 (+ 64us) service_pool.cc:225] Handling call
1026 02:56:37.438102 (+4538953us) ts_tablet_manager.cc:1151] Deleting tablet 3b77bb6b4722449c9699b548a9ef6314
1026 02:56:37.438115 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.438374 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4538707}
I20251026 02:56:37.440431 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 550) took 4473 ms. Trace:
I20251026 02:56:37.440584 2372 rpcz_store.cc:276] 1026 02:56:32.966576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.966634 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:37.440102 (+4473468us) ts_tablet_manager.cc:1151] Deleting tablet 6ea364c4a5cf4f3f99d9ff6904add697
1026 02:56:37.440147 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.440412 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4473274}
I20251026 02:56:37.441709 2372 tablet_replica.cc:333] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:37.441924 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3b77bb6b4722449c9699b548a9ef6314 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b77bb6b4722449c9699b548a9ef6314 already in progress: creating tablet
I20251026 02:56:37.442098 2372 raft_consensus.cc:2243] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:37.442281 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 6ea364c4a5cf4f3f99d9ff6904add697 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ea364c4a5cf4f3f99d9ff6904add697 already in progress: creating tablet
I20251026 02:56:37.442442 2372 raft_consensus.cc:2272] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:37.444294 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c861702df3424cdba2d706ea025a6529"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.444453 2378 tablet_bootstrap.cc:492] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:37.444391 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a4383df3509649cb824fea52dd98449e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.444725 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_dc184bc582ff43dc9e510f42ebe22815"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.444775 2378 ts_tablet_manager.cc:1403] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.274s user 0.004s sys 0.001s
W20251026 02:56:37.445219 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4383df3509649cb824fea52dd98449e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.445377 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_11f1980aeb6a4f38a5c9eebc431ddd86" } new_table_name: "default.Table_20d4fe3113a8478bb9284318fbe94aeb" modify_external_catalogs: true
W20251026 02:56:37.446431 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc184bc582ff43dc9e510f42ebe22815 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.446556 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_26f3ddff250042de9c91eff90955f4ab" } new_table_name: "default.Table_20aea86003f649d282631fee5df071c8" modify_external_catalogs: true
I20251026 02:56:37.446527 2378 raft_consensus.cc:359] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.446931 2378 raft_consensus.cc:385] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.447067 2378 raft_consensus.cc:740] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.447576 2378 consensus_queue.cc:260] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:37.449060 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c861702df3424cdba2d706ea025a6529 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.430799 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (table default.Table_1fa437ef7f89450bb6f672a1fb491baa [id=44c37945ff9f48e8b2c07648394ac177]). Expected version 1 got 0
I20251026 02:56:37.455364 1663 catalog_manager.cc:5649] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:37.455883 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (table default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5]). Expected version 1 got 0
I20251026 02:56:37.457415 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_dc22b2aa5fe64641b34fa498992594cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.458039 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc22b2aa5fe64641b34fa498992594cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.465559 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 7c11a71c28d646169d34f0a2fda5ac08 (DEFAULT_TABLE table=default.Table_7dc4340a68b546eba896325bdae7e3db [id=54d3e4993ce84e1b9d45f1e0832d04ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.466567 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c11a71c28d646169d34f0a2fda5ac08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.467483 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d2cc6b16d34331bb80c576c0cd2ce0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:37.467700 2120 raft_consensus.cc:2468] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:37.467862 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 9a345fa5edbd48d2891b8b40b09cb3f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a345fa5edbd48d2891b8b40b09cb3f8 already in progress: creating tablet
W20251026 02:56:37.468385 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet a3af2695c41a4439bc4092b7cb1b2edf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3af2695c41a4439bc4092b7cb1b2edf already in progress: creating tablet
W20251026 02:56:37.468801 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 already in progress: creating tablet
I20251026 02:56:37.469099 2372 ts_tablet_manager.cc:1916] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:37.469686 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:37.469842 2120 raft_consensus.cc:2468] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:37.470665 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 8aa0575af07e433c8d96acab31edb4af from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:37.471935 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_056ac0729a274fb19574d75c2f37511b" } new_table_name: "default.Table_c96a0fe4ea684dd29d1efeaab923e539" modify_external_catalogs: true
W20251026 02:56:37.472456 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 106524801a204a81b9f194321768f433 consensus state has no leader
I20251026 02:56:37.474350 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_adc311a814324b2db4e8aee6caa91f09" } new_table_name: "default.Table_28e24ecc433d4e0b8d7395abc286c54f" modify_external_catalogs: true
I20251026 02:56:37.485707 2381 tablet_bootstrap.cc:492] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:37.490818 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 7c11a71c28d646169d34f0a2fda5ac08 (DEFAULT_TABLE table=default.Table_7dc4340a68b546eba896325bdae7e3db [id=54d3e4993ce84e1b9d45f1e0832d04ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.494761 2381 ts_tablet_manager.cc:1403] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.097s user 0.002s sys 0.002s
I20251026 02:56:37.495560 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ed035c4e56264e5c8401998633ac2a04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.496315 2378 ts_tablet_manager.cc:1434] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.051s user 0.002s sys 0.001s
W20251026 02:56:37.496493 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed035c4e56264e5c8401998633ac2a04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.496852 2378 tablet_bootstrap.cc:492] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:37.498433 2381 raft_consensus.cc:359] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.498986 2381 raft_consensus.cc:385] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.499400 1915 leader_election.cc:304] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:37.499792 3318 raft_consensus.cc:2749] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.500170 2381 raft_consensus.cc:740] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.500909 2381 consensus_queue.cc:260] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.501940 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8aa0575af07e433c8d96acab31edb4af
W20251026 02:56:37.501932 1915 leader_election.cc:343] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: ca4611c0a494425ca50d599b310872c8
I20251026 02:56:37.502048 1915 leader_election.cc:304] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:37.502456 1661 catalog_manager.cc:6719] ReplaceTablet: tablet 8aa0575af07e433c8d96acab31edb4af deleted and replaced by tablet f2bae845251944d780b06be90cd659e5
I20251026 02:56:37.502442 1915 leader_election.cc:304] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:37.502691 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 6979455632e54d2081065954771f34d3 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:37.502779 3372 raft_consensus.cc:2749] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.502929 3372 raft_consensus.cc:2749] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.503163 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 8aa0575af07e433c8d96acab31edb4af with delete_type TABLET_DATA_DELETED (replaced by tablet f2bae845251944d780b06be90cd659e5) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.503410 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 8aa0575af07e433c8d96acab31edb4af with delete_type TABLET_DATA_DELETED (replaced by tablet f2bae845251944d780b06be90cd659e5) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:37.504004 1650 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet bd11b620e78c499ebaa1b35f361242c3 (table default.Table_381e082f8f9b4d589c346ba1879166bd [id=9158d553997949bfbe37cf153f6b1ab7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bd11b620e78c499ebaa1b35f361242c3 consensus state has no leader
W20251026 02:56:37.504489 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a43c7d16330644b5bfa0a88e5b624f6b has no consensus state
I20251026 02:56:37.505998 2381 ts_tablet_manager.cc:1434] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20251026 02:56:37.509207 2381 tablet_bootstrap.cc:492] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:37.513393 2376 ts_tablet_manager.cc:1929] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:37.513787 2376 log.cc:1199] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/b1a2522801364693be0bda6fa45a697e
I20251026 02:56:37.520475 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ab132fb4514e009ce6b32247291a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.520814 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c11a71c28d646169d34f0a2fda5ac08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.524435 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cd431fcd27c4d3db2ea3dc4c265755d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.525246 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 04d47a34ccdc48ff83445a6d5b07be94 (DEFAULT_TABLE table=default.Table_37d7dc70b2c44c8ea6a882e8bbb8c5d5 [id=4e071c7ac72545d49d618fbd26914741]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.525903 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04d47a34ccdc48ff83445a6d5b07be94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.526714 2372 ts_tablet_manager.cc:1929] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:37.526806 2372 log.cc:1199] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/5caeb11322fe48348ebb523967152187
I20251026 02:56:37.527117 2372 ts_tablet_manager.cc:1950] T 5caeb11322fe48348ebb523967152187 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:37.528508 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec94273925e94fb09f55249426bc4eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.534989 2378 tablet_bootstrap.cc:654] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.536008 2376 ts_tablet_manager.cc:1950] T b1a2522801364693be0bda6fa45a697e P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:37.536042 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b8b0b75eb394e9d9a91f51156195912. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.540356 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e424a9c89e64b1c98ada17fa5c31e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.545727 2375 tablet_bootstrap.cc:654] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.548542 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 520) took 4651 ms. Trace:
I20251026 02:56:37.548743 2376 rpcz_store.cc:276] 1026 02:56:32.901361 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.901409 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:36.943237 (+4041828us) ts_tablet_manager.cc:1151] Deleting tablet b1a2522801364693be0bda6fa45a697e
1026 02:56:36.943255 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.040152 (+ 96897us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.513052 (+472900us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.548517 (+ 35465us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4041500}
I20251026 02:56:37.549408 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet b1a2522801364693be0bda6fa45a697e (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]) successfully deleted
W20251026 02:56:37.552186 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4b0933f71cb4ac7a58986d16b613e64 consensus state has no leader
W20251026 02:56:37.552997 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b8b0b75eb394e9d9a91f51156195912 consensus state has no leader
I20251026 02:56:37.555934 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e3e8bac83d124aacb292bdec63fbb5fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.556751 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3e8bac83d124aacb292bdec63fbb5fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.556782 2377 ts_tablet_manager.cc:1929] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:37.556887 2377 log.cc:1199] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/115ccccf16b74efc9f24b607cf0c74d1
I20251026 02:56:37.557390 2377 ts_tablet_manager.cc:1950] T 115ccccf16b74efc9f24b607cf0c74d1 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:37.557832 2381 tablet_bootstrap.cc:654] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.558243 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_95c013f9c71247aeb254654850f8be47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.558388 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 509) took 6155 ms. Trace:
I20251026 02:56:37.558465 2377 rpcz_store.cc:276] 1026 02:56:31.402949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.403468 (+ 519us) service_pool.cc:225] Handling call
1026 02:56:37.237459 (+5833991us) ts_tablet_manager.cc:1151] Deleting tablet 115ccccf16b74efc9f24b607cf0c74d1
1026 02:56:37.237467 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.242245 (+ 4778us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.556656 (+314411us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.558364 (+ 1708us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5833685}
I20251026 02:56:37.558754 2377 tablet_replica.cc:333] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539: stopping tablet replica
W20251026 02:56:37.558816 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95c013f9c71247aeb254654850f8be47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.560477 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1b3d02c229434ae4a351fd493a20654b" } modify_external_catalogs: true
I20251026 02:56:37.560549 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 554) took 4529 ms. Trace:
I20251026 02:56:37.560670 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1b3d02c229434ae4a351fd493a20654b" } modify_external_catalogs: true
I20251026 02:56:37.560637 2372 rpcz_store.cc:276] 1026 02:56:33.030746 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.030791 (+ 45us) service_pool.cc:225] Handling call
1026 02:56:37.441624 (+4410833us) ts_tablet_manager.cc:1151] Deleting tablet 5caeb11322fe48348ebb523967152187
1026 02:56:37.441634 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.523697 (+ 82063us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.526622 (+ 2925us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.560520 (+ 33898us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4410662}
I20251026 02:56:37.560927 2372 tablet_replica.cc:333] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:37.561113 2372 raft_consensus.cc:2243] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:37.561259 2372 raft_consensus.cc:2272] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:37.562041 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e87acf02545f4f63987389f32903aec1" } new_table_name: "default.Table_2f1ecb7efe6244588067d3c2ba9be9ad" modify_external_catalogs: true
I20251026 02:56:37.562651 2372 ts_tablet_manager.cc:1916] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:37.562089 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 5caeb11322fe48348ebb523967152187 (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
I20251026 02:56:37.564561 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 04d47a34ccdc48ff83445a6d5b07be94 (DEFAULT_TABLE table=default.Table_37d7dc70b2c44c8ea6a882e8bbb8c5d5 [id=4e071c7ac72545d49d618fbd26914741]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.564929 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04d47a34ccdc48ff83445a6d5b07be94. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:37.565500 2053 leader_election.cc:343] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.565613 2053 leader_election.cc:304] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.565898 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0207b8f2a960407fa3a85a1280bb2256" } modify_external_catalogs: true
I20251026 02:56:37.566083 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0207b8f2a960407fa3a85a1280bb2256" } modify_external_catalogs: true
I20251026 02:56:37.566390 3272 raft_consensus.cc:2749] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:37.567337 2053 leader_election.cc:343] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.567441 2053 leader_election.cc:304] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.567775 3272 raft_consensus.cc:2749] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:37.568109 2053 leader_election.cc:343] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.568197 2053 leader_election.cc:304] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.568538 3272 raft_consensus.cc:2749] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.569195 1947 tablet_service.cc:1552] Processing DeleteTablet for tablet 863ce9e1adb444b7b1399b968ea79bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.569329 3318 consensus_queue.cc:1048] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:37.570442 2053 leader_election.cc:304] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.570907 3006 raft_consensus.cc:2804] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:37.570976 3006 raft_consensus.cc:493] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:37.571041 3006 raft_consensus.cc:3060] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.572154 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e250def75d2042ee89b9ce3bd7ed34ab" } modify_external_catalogs: true
I20251026 02:56:37.572513 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e250def75d2042ee89b9ce3bd7ed34ab" } modify_external_catalogs: true
I20251026 02:56:37.574095 3006 raft_consensus.cc:515] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.575022 3006 leader_election.cc:290] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.577025 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 521) took 4675 ms. Trace:
I20251026 02:56:37.577683 2376 rpcz_store.cc:276] 1026 02:56:32.901898 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.901948 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:37.568972 (+4667024us) ts_tablet_manager.cc:1151] Deleting tablet 3b77bb6b4722449c9699b548a9ef6314
1026 02:56:37.576841 (+ 7869us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.577000 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4666847}
I20251026 02:56:37.578105 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3327d26a2c2c4d0b86d98151f5940bd7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:37.578775 2053 leader_election.cc:343] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.579102 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 522) took 4609 ms. Trace:
I20251026 02:56:37.579233 2376 rpcz_store.cc:276] 1026 02:56:32.969239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.969302 (+ 63us) service_pool.cc:225] Handling call
1026 02:56:37.578349 (+4609047us) ts_tablet_manager.cc:1151] Deleting tablet 6ea364c4a5cf4f3f99d9ff6904add697
1026 02:56:37.578979 (+ 630us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.579067 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4608770}
I20251026 02:56:37.579720 2376 tablet_replica.cc:333] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:37.579964 2376 raft_consensus.cc:2243] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Raft consensus shutting down.
W20251026 02:56:37.580718 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3b77bb6b4722449c9699b548a9ef6314 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b77bb6b4722449c9699b548a9ef6314 already in progress: creating tablet
W20251026 02:56:37.581105 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 6ea364c4a5cf4f3f99d9ff6904add697 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ea364c4a5cf4f3f99d9ff6904add697 already in progress: creating tablet
I20251026 02:56:37.575011 2372 ts_tablet_manager.cc:1929] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:37.581784 2372 log.cc:1199] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/2524ee0c964345459c3e62f1e39b8f9e
I20251026 02:56:37.581845 3272 raft_consensus.cc:493] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.581995 3272 raft_consensus.cc:515] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.582109 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_bb11d644fbd64beb9783fd730f8bb577"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.582259 2372 ts_tablet_manager.cc:1950] T 2524ee0c964345459c3e62f1e39b8f9e P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:37.582592 3272 leader_election.cc:290] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:37.582753 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb11d644fbd64beb9783fd730f8bb577 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.584363 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 552) took 4553 ms. Trace:
I20251026 02:56:37.584452 2372 rpcz_store.cc:276] 1026 02:56:33.030547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.036548 (+ 6001us) service_pool.cc:225] Handling call
1026 02:56:37.560887 (+4524339us) ts_tablet_manager.cc:1151] Deleting tablet 2524ee0c964345459c3e62f1e39b8f9e
1026 02:56:37.560895 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.571842 (+ 10947us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.574911 (+ 3069us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.584341 (+ 9430us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1200,"tablet-delete.queue_time_us":4524138}
I20251026 02:56:37.584785 2372 tablet_replica.cc:333] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:37.584955 2372 raft_consensus.cc:2243] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:37.584934 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 2524ee0c964345459c3e62f1e39b8f9e (table default.Table_164cb1784f4645e499339f7163fdfc2b [id=91b4b1a8162441299c680e7c202a96d9]) successfully deleted
I20251026 02:56:37.585224 2372 raft_consensus.cc:2272] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:37.585777 2372 ts_tablet_manager.cc:1916] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:37.586288 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a6e8714ddccb4139adb6c8320e06500b" } modify_external_catalogs: true
I20251026 02:56:37.586511 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a6e8714ddccb4139adb6c8320e06500b" } modify_external_catalogs: true
W20251026 02:56:37.591243 2053 leader_election.cc:343] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.591389 2053 leader_election.cc:304] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.591761 3272 raft_consensus.cc:2749] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.594308 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 1a36b8a4beca483bb853d27961f5281a (DEFAULT_TABLE table=default.Table_96ed23975f9842f9a636936ad4f2727d [id=0e7597b5e7ce4b339035ff2ae3f069a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.594856 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a36b8a4beca483bb853d27961f5281a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.595993 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 6979455632e54d2081065954771f34d3
I20251026 02:56:37.596495 1661 catalog_manager.cc:6719] ReplaceTablet: tablet 6979455632e54d2081065954771f34d3 deleted and replaced by tablet 6294c8954f9749f998d5ae1c856e02ff
I20251026 02:56:37.596781 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 1370b9ad917344c093e5da3d0e9b1cf0 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:37.598209 2096 tablet_service.cc:1505] Processing CreateTablet for tablet bcd5621aa3754733b22508be89f080db (DEFAULT_TABLE table=default.Table_39b5d99fe09d427389ea14393b4f56ef [id=390e5f55c3d149129bcfd5a38f7e1b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.598590 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd5621aa3754733b22508be89f080db. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:37.600569 2053 leader_election.cc:343] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.600703 2053 leader_election.cc:304] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.601135 2381 tablet_bootstrap.cc:492] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:37.601123 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 92ae57c9a65a495e99e61efa123e8e88 (DEFAULT_TABLE table=default.Table_169fb347542948c896eb9cd54f60b8ff [id=0833ecbb5cba4e6cb41b7494590df155]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.601163 3272 raft_consensus.cc:2749] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.601274 2381 ts_tablet_manager.cc:1403] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.093s user 0.002s sys 0.002s
I20251026 02:56:37.601497 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92ae57c9a65a495e99e61efa123e8e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.602023 2381 raft_consensus.cc:359] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.602192 2381 raft_consensus.cc:385] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.602237 2381 raft_consensus.cc:740] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.602456 2381 consensus_queue.cc:260] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.603803 2381 ts_tablet_manager.cc:1434] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251026 02:56:37.605705 2053 leader_election.cc:343] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.605921 2053 leader_election.cc:304] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.606395 3272 raft_consensus.cc:2749] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.606833 2053 leader_election.cc:304] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.607302 3272 raft_consensus.cc:2804] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:37.607421 3272 raft_consensus.cc:493] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:37.607540 3272 raft_consensus.cc:3060] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.609288 2381 tablet_bootstrap.cc:492] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:37.611012 3272 raft_consensus.cc:515] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.611701 3272 leader_election.cc:290] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.612160 2381 tablet_bootstrap.cc:654] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.615418 2053 leader_election.cc:304] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.615887 3272 raft_consensus.cc:2804] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:37.616012 3272 raft_consensus.cc:493] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:37.616160 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 6979455632e54d2081065954771f34d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6294c8954f9749f998d5ae1c856e02ff) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:37.616394 2053 leader_election.cc:343] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.616569 2053 leader_election.cc:304] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.616685 2089 tablet_service.cc:1505] Processing CreateTablet for tablet b1f6e2c58c7f4367813eb66ebda5e0ca (DEFAULT_TABLE table=default.Table_7a07c4cc0b48429aad305ac537734fd5 [id=3c4691e3af784686bc7f96d650aae38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.616842 3006 raft_consensus.cc:2749] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.617127 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f6e2c58c7f4367813eb66ebda5e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.616148 3272 raft_consensus.cc:3060] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.619783 3272 raft_consensus.cc:515] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.583437 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa0aa6cdd0de4b0a91f553c2341e9c01" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:37.620930 2376 pending_rounds.cc:70] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8: Trying to abort 1 pending ops.
I20251026 02:56:37.621008 2376 pending_rounds.cc:77] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888534600396800 op_type: NO_OP noop_request { }
I20251026 02:56:37.621610 2376 raft_consensus.cc:2889] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:37.621724 2376 raft_consensus.cc:2272] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:37.622272 2376 ts_tablet_manager.cc:1916] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:37.590576 1962 tablet_service.cc:1505] Processing CreateTablet for tablet d8d60d9a610b4311b7a485ada922dae1 (DEFAULT_TABLE table=default.Table_d13a1abe2fae4061b799c01104b52dd5 [id=d5e08c87562d4daaa75b34412c1aa90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.597501 1947 tablet_service.cc:1505] Processing CreateTablet for tablet 1a36b8a4beca483bb853d27961f5281a (DEFAULT_TABLE table=default.Table_96ed23975f9842f9a636936ad4f2727d [id=0e7597b5e7ce4b339035ff2ae3f069a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.597654 1955 tablet_service.cc:1505] Processing CreateTablet for tablet bcd5621aa3754733b22508be89f080db (DEFAULT_TABLE table=default.Table_39b5d99fe09d427389ea14393b4f56ef [id=390e5f55c3d149129bcfd5a38f7e1b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.597833 1954 tablet_service.cc:1505] Processing CreateTablet for tablet b1f6e2c58c7f4367813eb66ebda5e0ca (DEFAULT_TABLE table=default.Table_7a07c4cc0b48429aad305ac537734fd5 [id=3c4691e3af784686bc7f96d650aae38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.598310 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 92ae57c9a65a495e99e61efa123e8e88 (DEFAULT_TABLE table=default.Table_169fb347542948c896eb9cd54f60b8ff [id=0833ecbb5cba4e6cb41b7494590df155]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.599087 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 6979455632e54d2081065954771f34d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6294c8954f9749f998d5ae1c856e02ff) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.623293 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8d60d9a610b4311b7a485ada922dae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.600548 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 0ccfd158d54641508138f481782c652f (DEFAULT_TABLE table=default.Table_03f92e9829754bccb37129e4415098d3 [id=17786ec363264bfc93db89115217b1f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.612373 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:37.623742 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f6e2c58c7f4367813eb66ebda5e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.625138 3239 raft_consensus.cc:493] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.625272 3239 raft_consensus.cc:515] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.625867 3239 leader_election.cc:290] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:37.626705 2195 leader_election.cc:304] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.626926 3239 raft_consensus.cc:2804] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:37.626991 3239 raft_consensus.cc:493] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:37.627048 3239 raft_consensus.cc:3060] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.627082 1980 raft_consensus.cc:3060] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.628150 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92ae57c9a65a495e99e61efa123e8e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.628762 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd5621aa3754733b22508be89f080db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.629545 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a36b8a4beca483bb853d27961f5281a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.629678 3239 raft_consensus.cc:515] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.630839 3239 leader_election.cc:290] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:37.632485 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ccfd158d54641508138f481782c652f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.636018 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet b1b6613742a24c21a1db6ebe858a846f from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:37.637475 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_47d5f6b5a01b4d4d991fca2b89af298e" } modify_external_catalogs: true
I20251026 02:56:37.637825 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_47d5f6b5a01b4d4d991fca2b89af298e" } modify_external_catalogs: true
W20251026 02:56:37.638492 2195 consensus_peers.cc:597] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4011930a277d410f863a1558416c4b0f. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:37.638705 2195 consensus_peers.cc:597] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63a63b796191465aab9a63d3ffb0554d. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:37.640801 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce922040298d4952af13652eeda7a234" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:37.641162 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be1c0948c0ae4563b7cb644521914cb9" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:37.641549 2194 consensus_peers.cc:597] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86d8e1e21c5f440d8baf76c357d01ad2. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:37.641783 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet d7313a6396c541d7ac6d8ca2e219af90
W20251026 02:56:37.642059 2194 leader_election.cc:343] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:37.642288 2194 leader_election.cc:343] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.643249 2381 tablet_bootstrap.cc:492] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:37.643473 2381 ts_tablet_manager.cc:1403] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.003s
I20251026 02:56:37.644603 2381 raft_consensus.cc:359] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.645162 3318 raft_consensus.cc:493] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.645324 3318 raft_consensus.cc:515] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.644918 2381 raft_consensus.cc:385] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.646481 2381 raft_consensus.cc:740] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.647063 2381 consensus_queue.cc:260] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.648659 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_138fc883cb304daeb0fc59f09921d7c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.648867 2381 ts_tablet_manager.cc:1434] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
W20251026 02:56:37.649242 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_138fc883cb304daeb0fc59f09921d7c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.649389 3318 leader_election.cc:290] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:37.650453 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet d7313a6396c541d7ac6d8ca2e219af90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:37.651062 1916 leader_election.cc:343] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 863ce9e1adb444b7b1399b968ea79bb4
I20251026 02:56:37.652531 2381 tablet_bootstrap.cc:492] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:37.654212 2381 tablet_bootstrap.cc:654] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:37.659823 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e645650709f47eea4243ab64ec31312 consensus state has no leader
I20251026 02:56:37.661803 3493 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.150.60:33729 every 8 connection(s)
W20251026 02:56:37.662564 2195 consensus_peers.cc:597] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e56b259a314422aba35eb129021bbeb. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:37.662220 3187 rpc_server.cc:307] RPC server started. Bound to: 127.1.150.60:33729
I20251026 02:56:37.666620 3187 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-2/data/info.pb
I20251026 02:56:37.668548 1624 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu as pid 3187
W20251026 02:56:37.620844 2053 leader_election.cc:343] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:37.676538 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 55695.5us
W20251026 02:56:37.677042 2053 leader_election.cc:343] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.646819 2375 tablet_bootstrap.cc:492] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:37.677469 2375 ts_tablet_manager.cc:1403] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.338s user 0.000s sys 0.005s
I20251026 02:56:37.678472 2375 raft_consensus.cc:359] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.678714 2375 raft_consensus.cc:385] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.678812 2375 raft_consensus.cc:740] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.679097 2375 consensus_queue.cc:260] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.684368 2375 ts_tablet_manager.cc:1434] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251026 02:56:37.685055 2375 tablet_bootstrap.cc:492] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:37.693996 2082 tablet_service.cc:1505] Processing CreateTablet for tablet d8d60d9a610b4311b7a485ada922dae1 (DEFAULT_TABLE table=default.Table_d13a1abe2fae4061b799c01104b52dd5 [id=d5e08c87562d4daaa75b34412c1aa90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.694419 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8d60d9a610b4311b7a485ada922dae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.695537 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a378e913112c443db869016e3b98fd0e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:37.695647 2377 raft_consensus.cc:2243] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:37.695866 2377 raft_consensus.cc:2272] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:37.696349 2053 leader_election.cc:343] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.696373 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet d7313a6396c541d7ac6d8ca2e219af90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.698701 2378 tablet_bootstrap.cc:492] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:37.698820 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e6acd68207564173ac53389222b0c621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.699084 2378 ts_tablet_manager.cc:1403] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.202s user 0.015s sys 0.015s
W20251026 02:56:37.699416 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6acd68207564173ac53389222b0c621 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.700042 3272 leader_election.cc:290] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.700891 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_888cc00019e343279dac7386f384975b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:37.701061 2381 tablet_bootstrap.cc:492] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:37.701211 2381 ts_tablet_manager.cc:1403] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.010s
W20251026 02:56:37.701651 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_888cc00019e343279dac7386f384975b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.701933 2381 raft_consensus.cc:359] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.702106 2381 raft_consensus.cc:385] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.702173 2381 raft_consensus.cc:740] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.702394 2381 consensus_queue.cc:260] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.702864 2381 ts_tablet_manager.cc:1434] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:37.703164 2381 tablet_bootstrap.cc:492] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:37.703806 2377 ts_tablet_manager.cc:1916] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:37.709626 2378 raft_consensus.cc:359] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.709822 2378 raft_consensus.cc:385] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.709878 2378 raft_consensus.cc:740] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.710078 2378 consensus_queue.cc:260] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.710724 2378 ts_tablet_manager.cc:1434] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251026 02:56:37.711000 2378 tablet_bootstrap.cc:492] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:37.711433 2375 tablet_bootstrap.cc:654] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.716147 3239 raft_consensus.cc:493] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.716394 3239 raft_consensus.cc:515] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.705227 2381 tablet_bootstrap.cc:654] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.717470 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741792cf96164695b03de3fb86b12602" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:37.717969 2194 leader_election.cc:343] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:37.705492 2195 consensus_peers.cc:597] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fe0e06a86e241bc91a1db4219fe25bc. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:37.718147 3366 raft_consensus.cc:493] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.718259 3366 raft_consensus.cc:515] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.718755 3366 leader_election.cc:290] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:37.719370 3239 leader_election.cc:290] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:37.720795 2195 consensus_peers.cc:597] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 863ce9e1adb444b7b1399b968ea79bb4. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:37.720955 2195 leader_election.cc:304] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.721660 3444 raft_consensus.cc:2804] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:37.721731 3444 raft_consensus.cc:493] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:37.721786 3444 raft_consensus.cc:3060] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.724563 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "046a34f9408b406ba08471421f422064" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:37.725090 2194 leader_election.cc:343] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.727312 3444 raft_consensus.cc:515] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.728044 3444 leader_election.cc:290] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:37.728461 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d2cc6b16d34331bb80c576c0cd2ce0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:37.729956 2194 leader_election.cc:343] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.733043 2372 ts_tablet_manager.cc:1929] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:37.734120 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0ddeba351c264f67b75b66c55c9734b5
I20251026 02:56:37.734583 2195 leader_election.cc:304] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.734819 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ddeba351c264f67b75b66c55c9734b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.735105 3444 raft_consensus.cc:2804] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:37.735332 3444 raft_consensus.cc:493] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:37.735357 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 63ebb91695ec495cbf44803a0769d19e (DEFAULT_TABLE table=default.Table_7cb8287939ea4ce3817a6043c996784e [id=8035d53b0b4747039e79b1c6f7a9f1b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.735445 3444 raft_consensus.cc:3060] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.738399 3444 raft_consensus.cc:515] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.739912 3444 leader_election.cc:290] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:37.742053 2378 tablet_bootstrap.cc:654] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.742524 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:37.743083 2194 leader_election.cc:343] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.745659 3318 raft_consensus.cc:493] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.745792 3318 raft_consensus.cc:515] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.746359 3318 leader_election.cc:290] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:37.749974 2195 consensus_peers.cc:597] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ced7ad6a601d432cbc13b3199a0506e4. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:37.750699 2243 tablet_service.cc:1505] Processing CreateTablet for tablet ec94273925e94fb09f55249426bc4eaa (DEFAULT_TABLE table=default.Table_8248296e6b4a45d497271230d96ac519 [id=4caa9908fba349ef9ed1da38ac8fbe7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.750748 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet ba08469339184b11b9ced9e6bdd63d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.751139 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec94273925e94fb09f55249426bc4eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.751179 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 22e4a433801a4cb58062d3e76c760b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.751521 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet a111848706b743c9beb2fbd21f59d7f3 with delete_type TABLET_DATA_DELETED (replaced by tablet d7313a6396c541d7ac6d8ca2e219af90) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.751533 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.751835 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet ce922040298d4952af13652eeda7a234 with delete_type TABLET_DATA_DELETED (replaced by tablet e830f1aa1270420c9b28a0491a1a0960) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.752166 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 3e424a9c89e64b1c98ada17fa5c31e9e (DEFAULT_TABLE table=default.Table_b12aefa4eb5b4380a6b9a31290cd24dc [id=1a4bff2fe09e4c2588c2ede9c8d95757]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.752264 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 47037db3602246a2ab6224557286162f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.752549 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e424a9c89e64b1c98ada17fa5c31e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.752628 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 17d2cc6b16d34331bb80c576c0cd2ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.753335 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 44af74033ccf4d048475f12d164938f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.753664 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet a378e913112c443db869016e3b98fd0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.754123 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 4b5e61afe90e4bc4a2897de21e0b3919 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:37.754858 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b12aefa4eb5b4380a6b9a31290cd24dc" } new_table_name: "default.Table_a22a7510b6a249398d85c3fac300f90a" modify_external_catalogs: true
I20251026 02:56:37.756256 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 3327d26a2c2c4d0b86d98151f5940bd7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7177931353164b2aa82c00b2807ccd63) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.756557 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 8aa0575af07e433c8d96acab31edb4af with delete_type TABLET_DATA_DELETED (replaced by tablet f2bae845251944d780b06be90cd659e5) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.756510 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 0b8b0b75eb394e9d9a91f51156195912 (DEFAULT_TABLE table=default.Table_adc311a814324b2db4e8aee6caa91f09 [id=418104d67c844ac19ad891922155dd86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.756683 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet b8cad7adae7e43469ceadbedeee53564 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.756723 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 7c11a71c28d646169d34f0a2fda5ac08 (DEFAULT_TABLE table=default.Table_7dc4340a68b546eba896325bdae7e3db [id=54d3e4993ce84e1b9d45f1e0832d04ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.756848 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 6108219254bd448c852096c0c967384b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.756922 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b8b0b75eb394e9d9a91f51156195912. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.757073 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet cfc3aed64ad0439cb22fd6815c973d87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b77bb6b4722449c9699b548a9ef6314) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.757247 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f5cd37c87f34a40bf05829af6de0473 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.760131 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c11a71c28d646169d34f0a2fda5ac08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.761624 2234 tablet_service.cc:1505] Processing CreateTablet for tablet b1f6e2c58c7f4367813eb66ebda5e0ca (DEFAULT_TABLE table=default.Table_7a07c4cc0b48429aad305ac537734fd5 [id=3c4691e3af784686bc7f96d650aae38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.761937 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f6e2c58c7f4367813eb66ebda5e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.761886 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 1a36b8a4beca483bb853d27961f5281a (DEFAULT_TABLE table=default.Table_96ed23975f9842f9a636936ad4f2727d [id=0e7597b5e7ce4b339035ff2ae3f069a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.762145 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a36b8a4beca483bb853d27961f5281a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.761574 2230 tablet_service.cc:1505] Processing CreateTablet for tablet d8d60d9a610b4311b7a485ada922dae1 (DEFAULT_TABLE table=default.Table_d13a1abe2fae4061b799c01104b52dd5 [id=d5e08c87562d4daaa75b34412c1aa90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.762403 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8d60d9a610b4311b7a485ada922dae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.763490 2226 tablet_service.cc:1505] Processing CreateTablet for tablet 04d47a34ccdc48ff83445a6d5b07be94 (DEFAULT_TABLE table=default.Table_37d7dc70b2c44c8ea6a882e8bbb8c5d5 [id=4e071c7ac72545d49d618fbd26914741]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.763895 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04d47a34ccdc48ff83445a6d5b07be94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.767755 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet be1c0948c0ae4563b7cb644521914cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.786394 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 0b04c71ba1cd429d90ff8e451ed45870 (DEFAULT_TABLE table=default.Table_d08c9c1fe775483492716153dba0072d [id=baef8f3df09649bf8bb2d9c2070c25d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.791792 2377 ts_tablet_manager.cc:1929] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:37.791940 2377 log.cc:1199] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/e62064fbdffb4954a3a1227efd6327ee
I20251026 02:56:37.795455 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 6979455632e54d2081065954771f34d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6294c8954f9749f998d5ae1c856e02ff) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.796361 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 92ae57c9a65a495e99e61efa123e8e88 (DEFAULT_TABLE table=default.Table_169fb347542948c896eb9cd54f60b8ff [id=0833ecbb5cba4e6cb41b7494590df155]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.797159 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92ae57c9a65a495e99e61efa123e8e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.796780 2241 tablet_service.cc:1505] Processing CreateTablet for tablet bcd5621aa3754733b22508be89f080db (DEFAULT_TABLE table=default.Table_39b5d99fe09d427389ea14393b4f56ef [id=390e5f55c3d149129bcfd5a38f7e1b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.803706 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd5621aa3754733b22508be89f080db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.804003 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63ebb91695ec495cbf44803a0769d19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.804289 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b04c71ba1cd429d90ff8e451ed45870. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.804641 1954 tablet_service.cc:1505] Processing CreateTablet for tablet d0b0954ff4824ee7b47dce18bdc61122 (DEFAULT_TABLE table=default.Table_2061f39f1e524edb8d9824942b2ef13d [id=78d00b5e85b6423ab7ada1d35df14f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.804958 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b0954ff4824ee7b47dce18bdc61122. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.808990 2378 tablet_bootstrap.cc:492] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:37.809137 2378 ts_tablet_manager.cc:1403] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.098s user 0.006s sys 0.002s
I20251026 02:56:37.809826 2378 raft_consensus.cc:359] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.809993 2378 raft_consensus.cc:385] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.810068 2378 raft_consensus.cc:740] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.810262 2378 consensus_queue.cc:260] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.810918 2378 ts_tablet_manager.cc:1434] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:37.811244 2378 tablet_bootstrap.cc:492] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:37.812902 2378 tablet_bootstrap.cc:654] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.813436 2376 ts_tablet_manager.cc:1929] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:37.813549 2376 log.cc:1199] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/e62064fbdffb4954a3a1227efd6327ee
I20251026 02:56:37.813869 2376 ts_tablet_manager.cc:1950] T e62064fbdffb4954a3a1227efd6327ee P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:37.761556 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9a5670a9e10c4a9db64881e7e14e839e
I20251026 02:56:37.817114 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a5670a9e10c4a9db64881e7e14e839e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.734563 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 0ccfd158d54641508138f481782c652f (DEFAULT_TABLE table=default.Table_03f92e9829754bccb37129e4415098d3 [id=17786ec363264bfc93db89115217b1f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.822378 2372 log.cc:1199] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/0de06ea2b0544402a979d9ddc487860f
I20251026 02:56:37.822329 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 8f3ae3a377af46449c3faca339f86053 (table default.Table_42e3536941484e83a992e13086dc316e [id=31f977e55ba041f59bcc721d469746fd]). Expected version 2 got 0
I20251026 02:56:37.822539 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 046a34f9408b406ba08471421f422064 (table default.Table_760b799491d7426a86aa1dc3c76fab31 [id=bc8a2d95ed134fae9f3e9deeac4f130a]). Expected version 3 got 0
I20251026 02:56:37.822644 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ccfd158d54641508138f481782c652f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.822752 2372 ts_tablet_manager.cc:1950] T 0de06ea2b0544402a979d9ddc487860f P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:37.787391 1658 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 352a284902c94f088e2e6ae006162db0 (table default.Table_6cd4ee82e9ec406f971669b7f0722677 [id=414375003cd44b5c8c9f0249b339c28a]). Expected version 1 got 0
I20251026 02:56:37.793678 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8688a0bf946841deb5d88c931538b129
I20251026 02:56:37.824627 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 8688a0bf946841deb5d88c931538b129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.824054 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e9df6f46f0f3408d9aa07452b0d7e77d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.825177 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9df6f46f0f3408d9aa07452b0d7e77d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.826532 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7ba3891cfe0b46329d460a7a3881d8d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.827034 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ba3891cfe0b46329d460a7a3881d8d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.827365 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 8688a0bf946841deb5d88c931538b129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.829080 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 9d12453c4225455881336b9f31d8e378 (DEFAULT_TABLE table=default.Table_a70ab000815d4d91b69ae8a6727c6b88 [id=e54f35a539db4b2ba5e7cec638df29e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.829289 2381 tablet_bootstrap.cc:492] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:37.829464 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d12453c4225455881336b9f31d8e378. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.829496 2381 ts_tablet_manager.cc:1403] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.126s user 0.012s sys 0.004s
I20251026 02:56:37.830391 2381 raft_consensus.cc:359] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.736187 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ddeba351c264f67b75b66c55c9734b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.746917 2113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352a284902c94f088e2e6ae006162db0" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:37.830768 2381 raft_consensus.cc:385] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.830837 2381 raft_consensus.cc:740] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.750891 2089 tablet_service.cc:1552] Processing DeleteTablet for tablet b1ca5b8379c94deab1130bfe8bdc87b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.831068 2381 consensus_queue.cc:260] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.831524 2381 ts_tablet_manager.cc:1434] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:37.832010 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 741792cf96164695b03de3fb86b12602 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.779122 2375 tablet_bootstrap.cc:492] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:37.792593 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 63ebb91695ec495cbf44803a0769d19e (DEFAULT_TABLE table=default.Table_7cb8287939ea4ce3817a6043c996784e [id=8035d53b0b4747039e79b1c6f7a9f1b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.832438 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet d7313a6396c541d7ac6d8ca2e219af90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.832531 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63ebb91695ec495cbf44803a0769d19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.833021 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 0ccfd158d54641508138f481782c652f (DEFAULT_TABLE table=default.Table_03f92e9829754bccb37129e4415098d3 [id=17786ec363264bfc93db89115217b1f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.833388 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 63ebb91695ec495cbf44803a0769d19e (DEFAULT_TABLE table=default.Table_7cb8287939ea4ce3817a6043c996784e [id=8035d53b0b4747039e79b1c6f7a9f1b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.833842 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 0b04c71ba1cd429d90ff8e451ed45870 (DEFAULT_TABLE table=default.Table_d08c9c1fe775483492716153dba0072d [id=baef8f3df09649bf8bb2d9c2070c25d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.804243 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 0b04c71ba1cd429d90ff8e451ed45870 (DEFAULT_TABLE table=default.Table_d08c9c1fe775483492716153dba0072d [id=baef8f3df09649bf8bb2d9c2070c25d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.836524 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b04c71ba1cd429d90ff8e451ed45870. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.836526 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a5670a9e10c4a9db64881e7e14e839e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.836910 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 8688a0bf946841deb5d88c931538b129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.839882 2237 tablet_service.cc:1505] Processing CreateTablet for tablet d0b0954ff4824ee7b47dce18bdc61122 (DEFAULT_TABLE table=default.Table_2061f39f1e524edb8d9824942b2ef13d [id=78d00b5e85b6423ab7ada1d35df14f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.817173 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a5670a9e10c4a9db64881e7e14e839e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.841641 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be1c0948c0ae4563b7cb644521914cb9" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:37.841658 2115 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce922040298d4952af13652eeda7a234" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:37.841843 2115 raft_consensus.cc:3060] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.841859 2120 raft_consensus.cc:2468] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:37.821281 2099 tablet_service.cc:1505] Processing CreateTablet for tablet d0b0954ff4824ee7b47dce18bdc61122 (DEFAULT_TABLE table=default.Table_2061f39f1e524edb8d9824942b2ef13d [id=78d00b5e85b6423ab7ada1d35df14f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.842512 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b0954ff4824ee7b47dce18bdc61122. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.845115 2115 raft_consensus.cc:2468] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:37.845942 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 115ccccf16b74efc9f24b607cf0c74d1 (table default.Table_7b6687267a1a4b2da7e02949e7b4c081 [id=ad981d3883cc4ee5b77defff9651ef31]) successfully deleted
I20251026 02:56:37.846781 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "046a34f9408b406ba08471421f422064" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:37.846963 2117 raft_consensus.cc:2468] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:37.846972 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "634d913c98c842a49d97a54f23752d26" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.847505 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741792cf96164695b03de3fb86b12602" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:37.847635 2114 raft_consensus.cc:2468] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:37.847726 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "863ce9e1adb444b7b1399b968ea79bb4" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.847831 2107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d2cc6b16d34331bb80c576c0cd2ce0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:37.847950 2107 raft_consensus.cc:3060] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.848106 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352a284902c94f088e2e6ae006162db0" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.848495 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dbe8002df6c4b64b8ec8260ca6b9cf5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.848712 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1370b9ad917344c093e5da3d0e9b1cf0
I20251026 02:56:37.848836 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff242ef9541a42d793c4484190956da6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.849171 1661 catalog_manager.cc:6719] ReplaceTablet: tablet 1370b9ad917344c093e5da3d0e9b1cf0 deleted and replaced by tablet 28f529b6c18d412e8caa10753eab4e01
I20251026 02:56:37.850287 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3327d26a2c2c4d0b86d98151f5940bd7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:37.850448 2257 raft_consensus.cc:3060] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.852070 2107 raft_consensus.cc:2468] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:37.852322 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 1370b9ad917344c093e5da3d0e9b1cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f529b6c18d412e8caa10753eab4e01) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.853772 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:37.853953 2120 raft_consensus.cc:2393] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 696476265495492790739ec126a08539 in current term 1: Already voted for candidate 0b6b0a6c4e144982b67980118cddba48 in this term.
I20251026 02:56:37.854403 2257 raft_consensus.cc:2468] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:37.855233 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ccfd158d54641508138f481782c652f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.855233 2381 tablet_bootstrap.cc:492] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:37.858743 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a70ab000815d4d91b69ae8a6727c6b88" } new_table_name: "default.Table_e1085a562ca74f1bb56e6661271a3b50" modify_external_catalogs: true
I20251026 02:56:37.859386 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63ebb91695ec495cbf44803a0769d19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.864285 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b04c71ba1cd429d90ff8e451ed45870. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.867964 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b1b6613742a24c21a1db6ebe858a846f
I20251026 02:56:37.868356 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 1370b9ad917344c093e5da3d0e9b1cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f529b6c18d412e8caa10753eab4e01) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.868472 1659 catalog_manager.cc:6719] ReplaceTablet: tablet b1b6613742a24c21a1db6ebe858a846f deleted and replaced by tablet 0c60d8227eed417595ef68d44c68ebcf
I20251026 02:56:37.868723 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b0954ff4824ee7b47dce18bdc61122. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.868824 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b7f76ac1487446019420d43442bcd4ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.869398 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7f76ac1487446019420d43442bcd4ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.870689 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet b1b6613742a24c21a1db6ebe858a846f with delete_type TABLET_DATA_DELETED (replaced by tablet 0c60d8227eed417595ef68d44c68ebcf) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.883020 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 63a63b796191465aab9a63d3ffb0554d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.883520 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet b1b6613742a24c21a1db6ebe858a846f with delete_type TABLET_DATA_DELETED (replaced by tablet 0c60d8227eed417595ef68d44c68ebcf) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.885677 2375 ts_tablet_manager.cc:1403] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.201s user 0.003s sys 0.001s
I20251026 02:56:37.886442 2375 raft_consensus.cc:359] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.886651 2375 raft_consensus.cc:385] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.886716 2375 raft_consensus.cc:740] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.886987 2375 consensus_queue.cc:260] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.887475 2375 ts_tablet_manager.cc:1434] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 02:56:37.890031 2375 tablet_bootstrap.cc:492] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:37.892050 2027 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:37.897272 2375 tablet_bootstrap.cc:654] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.898969 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa0aa6cdd0de4b0a91f553c2341e9c01" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:37.899679 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:37.899797 2378 tablet_bootstrap.cc:492] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:37.899870 2257 raft_consensus.cc:2393] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b6b0a6c4e144982b67980118cddba48 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:37.899920 2378 ts_tablet_manager.cc:1403] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.000s
I20251026 02:56:37.900750 2378 raft_consensus.cc:359] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.900930 2378 raft_consensus.cc:385] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.900996 2378 raft_consensus.cc:740] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.901228 2378 consensus_queue.cc:260] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.901751 2378 ts_tablet_manager.cc:1434] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:37.903927 2378 tablet_bootstrap.cc:492] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:37.905704 2378 tablet_bootstrap.cc:654] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.905797 3508 tablet.cc:1722] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251026 02:56:37.916486 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a378e913112c443db869016e3b98fd0e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:37.916689 2257 raft_consensus.cc:3060] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.917662 1915 leader_election.cc:304] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:37.918236 3318 raft_consensus.cc:2749] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.918699 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ddeba351c264f67b75b66c55c9734b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.919029 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 9d12453c4225455881336b9f31d8e378 (DEFAULT_TABLE table=default.Table_a70ab000815d4d91b69ae8a6727c6b88 [id=e54f35a539db4b2ba5e7cec638df29e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.919538 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d12453c4225455881336b9f31d8e378. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.923918 2378 tablet_bootstrap.cc:492] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:37.924063 2378 ts_tablet_manager.cc:1403] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.001s
I20251026 02:56:37.922484 2257 raft_consensus.cc:2468] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:37.925004 2378 raft_consensus.cc:359] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.925186 2378 raft_consensus.cc:385] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.925246 2378 raft_consensus.cc:740] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.925316 1915 leader_election.cc:304] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:37.925449 2378 consensus_queue.cc:260] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.925674 3372 raft_consensus.cc:2749] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:37.929387 2053 leader_election.cc:343] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.929522 2053 leader_election.cc:304] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.929708 2308 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
W20251026 02:56:37.929903 2053 leader_election.cc:343] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.929996 2053 leader_election.cc:304] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.930330 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3aa7a90a29ec430286786bf8aa9d4c02
I20251026 02:56:37.930497 3272 raft_consensus.cc:2749] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.930639 3272 raft_consensus.cc:2749] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.931677 1915 leader_election.cc:304] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:37.932094 3372 raft_consensus.cc:2749] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.933137 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_76d502990a87496d9d6d2d163e712939" } new_table_name: "default.Table_2bb5a8427c5044cc8dc28649dea1f4b1" modify_external_catalogs: true
I20251026 02:56:37.934136 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 3aa7a90a29ec430286786bf8aa9d4c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.934437 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 9d12453c4225455881336b9f31d8e378 (DEFAULT_TABLE table=default.Table_a70ab000815d4d91b69ae8a6727c6b88 [id=e54f35a539db4b2ba5e7cec638df29e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:37.934721 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_424693529d0d4c829755adc2c9835c0e" } new_table_name: "default.Table_66d05ce30ff642b58cc2d6658d509575" modify_external_catalogs: true
I20251026 02:56:37.935362 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d12453c4225455881336b9f31d8e378. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.935748 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet a08efd37968846d2958e54831488235c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.937819 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 3aa7a90a29ec430286786bf8aa9d4c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.939014 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 553) took 4908 ms. Trace:
I20251026 02:56:37.939277 2378 ts_tablet_manager.cc:1434] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20251026 02:56:37.939172 2372 rpcz_store.cc:276] 1026 02:56:33.030646 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.036795 (+ 6149us) service_pool.cc:225] Handling call
1026 02:56:37.584744 (+4547949us) ts_tablet_manager.cc:1151] Deleting tablet 0de06ea2b0544402a979d9ddc487860f
1026 02:56:37.584753 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.729506 (+144753us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.732937 (+ 3431us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.938993 (+206056us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4547799}
I20251026 02:56:37.940147 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 0de06ea2b0544402a979d9ddc487860f (table default.Table_6546379855b749b889279973db1820cd [id=b8be7adba1cd48ae979de506d892cc3d]) successfully deleted
I20251026 02:56:37.940583 2372 tablet_replica.cc:333] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:37.940846 2372 raft_consensus.cc:2243] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:37.941102 2372 raft_consensus.cc:2272] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:37.939136 2194 consensus_peers.cc:597] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:37.943665 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 3aa7a90a29ec430286786bf8aa9d4c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.943831 2378 tablet_bootstrap.cc:492] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:37.936062 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 1370b9ad917344c093e5da3d0e9b1cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f529b6c18d412e8caa10753eab4e01) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.944572 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet b1b6613742a24c21a1db6ebe858a846f with delete_type TABLET_DATA_DELETED (replaced by tablet 0c60d8227eed417595ef68d44c68ebcf) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:37.946486 2308 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:37.934139 2195 leader_election.cc:304] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.953745 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c6376c7fc54d43c68859ff371a420894"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.954368 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6376c7fc54d43c68859ff371a420894 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.956614 3328 raft_consensus.cc:2804] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:37.956702 3328 raft_consensus.cc:493] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:37.956777 3328 raft_consensus.cc:3060] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:37.957774 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet b1ca5b8379c94deab1130bfe8bdc87b0 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:37.958220 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 882fe81da21344ed99ea735fa79f73c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:37.958364 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0e0f079c121342b9bb342f26e3c24518"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.958884 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e0f079c121342b9bb342f26e3c24518 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.959829 3328 raft_consensus.cc:515] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.963332 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 5caeb11322fe48348ebb523967152187 with delete_type TABLET_DATA_DELETED (replaced by tablet d43b6f22dd4f4f9c9657f21514550312) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:37.965917 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be1c0948c0ae4563b7cb644521914cb9" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:37.967496 2378 tablet_bootstrap.cc:654] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:37.968447 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e424a9c89e64b1c98ada17fa5c31e9e consensus state has no leader
I20251026 02:56:37.972067 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_bd39192739b24c9ba278c4706acb20bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:37.972682 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd39192739b24c9ba278c4706acb20bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:37.973796 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1a48da091e984e09aaffb9caed79f2c1" } modify_external_catalogs: true
I20251026 02:56:37.973860 2053 leader_election.cc:304] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.974022 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1a48da091e984e09aaffb9caed79f2c1" } modify_external_catalogs: true
W20251026 02:56:37.974367 2053 leader_election.cc:343] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.974419 3006 raft_consensus.cc:2804] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:37.974543 2053 leader_election.cc:304] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.974586 3006 raft_consensus.cc:697] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:37.974797 3006 consensus_queue.cc:237] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.975310 3510 raft_consensus.cc:2749] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.982295 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 523) took 4651 ms. Trace:
I20251026 02:56:37.984726 2372 ts_tablet_manager.cc:1916] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:37.985281 2376 rpcz_store.cc:276] 1026 02:56:33.331020 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.333287 (+ 2267us) service_pool.cc:225] Handling call
1026 02:56:37.579676 (+4246389us) ts_tablet_manager.cc:1151] Deleting tablet e62064fbdffb4954a3a1227efd6327ee
1026 02:56:37.579684 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.713582 (+133898us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.813336 (+ 99754us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.982268 (+168932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4246115}
I20251026 02:56:37.986656 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet e62064fbdffb4954a3a1227efd6327ee (table default.Table_e9da6904da2c4d15bb17df7e9bbb3221 [id=fb1dbbc49ad94d85925be7fc850041e0]) successfully deleted
I20251026 02:56:37.990716 3372 raft_consensus.cc:493] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:37.990752 2053 leader_election.cc:304] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:37.990854 3372 raft_consensus.cc:515] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:37.991168 3496 sys_catalog.cc:263] Verifying existing consensus state
I20251026 02:56:37.991243 3510 raft_consensus.cc:2749] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 02:56:37.991503 3372 leader_election.cc:290] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:37.992714 2378 tablet_bootstrap.cc:492] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:37.992830 2378 ts_tablet_manager.cc:1403] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.001s
I20251026 02:56:37.993594 2378 raft_consensus.cc:359] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.993762 2378 raft_consensus.cc:385] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:37.993824 2378 raft_consensus.cc:740] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:37.994071 2378 consensus_queue.cc:260] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:37.994594 2378 ts_tablet_manager.cc:1434] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:37.994760 2165 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
W20251026 02:56:37.995596 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d12453c4225455881336b9f31d8e378 consensus state has no leader
W20251026 02:56:37.997017 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09f5265d85c6496d85c555b2b847373a consensus state has no leader
I20251026 02:56:37.997442 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_70b9f6803b9345bc8010f0e216bff485" } new_table_name: "default.Table_cf4027782c3841c58cea284393344525" modify_external_catalogs: true
I20251026 02:56:37.998124 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0ea3c4e29f264b34b9f6030b92262af6" } modify_external_catalogs: true
I20251026 02:56:37.998296 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0ea3c4e29f264b34b9f6030b92262af6" } modify_external_catalogs: true
W20251026 02:56:37.999634 1916 leader_election.cc:343] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 5caeb11322fe48348ebb523967152187
I20251026 02:56:38.000530 2376 tablet_replica.cc:333] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:38.000720 2376 raft_consensus.cc:2243] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:38.001048 2376 pending_rounds.cc:70] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8: Trying to abort 1 pending ops.
I20251026 02:56:38.001106 2376 pending_rounds.cc:77] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888531030032384 op_type: NO_OP noop_request { }
I20251026 02:56:38.001199 2376 raft_consensus.cc:2889] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:38.001268 2376 raft_consensus.cc:2272] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.001801 2376 ts_tablet_manager.cc:1916] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:37.962428 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a263a68d6e4b5d8d35c78595266e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:37.965770 3328 leader_election.cc:290] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:37.966480 2194 leader_election.cc:343] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:37.966532 2195 leader_election.cc:304] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.008803 2195 leader_election.cc:304] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.009135 2195 leader_election.cc:304] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.009301 3366 raft_consensus.cc:2804] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 02:56:38.009477 2195 consensus_peers.cc:597] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 634d913c98c842a49d97a54f23752d26. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:38.009624 2195 leader_election.cc:304] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.009761 3444 raft_consensus.cc:2804] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:38.009845 3444 raft_consensus.cc:493] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:38.009912 3444 raft_consensus.cc:3060] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.010574 3239 raft_consensus.cc:2804] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:38.010644 3239 raft_consensus.cc:493] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:38.010699 3239 raft_consensus.cc:3060] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.012696 1980 raft_consensus.cc:3060] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.012694 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a263fecc382f46039c6cdad42a49517d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.013316 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a263fecc382f46039c6cdad42a49517d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.014844 3328 raft_consensus.cc:2749] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.015900 2375 tablet_bootstrap.cc:492] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:38.016022 2375 ts_tablet_manager.cc:1403] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.126s user 0.002s sys 0.003s
I20251026 02:56:38.016713 3239 raft_consensus.cc:515] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.017012 2378 tablet_bootstrap.cc:492] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:38.017737 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741792cf96164695b03de3fb86b12602" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:38.018666 2378 tablet_bootstrap.cc:654] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:38.019908 2194 leader_election.cc:343] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.020757 3239 leader_election.cc:290] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:38.022612 2375 raft_consensus.cc:359] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.022778 2375 raft_consensus.cc:385] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.022822 2375 raft_consensus.cc:740] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.023048 2375 consensus_queue.cc:260] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.023633 2375 ts_tablet_manager.cc:1434] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251026 02:56:38.029357 3444 raft_consensus.cc:515] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.030238 3444 leader_election.cc:290] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:38.032852 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 276bd3b97c97426daf802e1f57feb265 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.035429 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 1a36b8a4beca483bb853d27961f5281a from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:37.983979 2381 tablet_bootstrap.cc:654] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:37.984119 2377 ts_tablet_manager.cc:1950] T e62064fbdffb4954a3a1227efd6327ee P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:38.042793 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet aa0aa6cdd0de4b0a91f553c2341e9c01
I20251026 02:56:38.043608 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 510) took 6640 ms. Trace:
I20251026 02:56:38.043686 2377 rpcz_store.cc:276] 1026 02:56:31.403096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.403827 (+ 731us) service_pool.cc:225] Handling call
1026 02:56:37.558710 (+6154883us) ts_tablet_manager.cc:1151] Deleting tablet e62064fbdffb4954a3a1227efd6327ee
1026 02:56:37.558718 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:37.788755 (+230037us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:37.791683 (+ 2928us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.043589 (+251906us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6154694}
I20251026 02:56:38.043988 2377 tablet_replica.cc:333] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:38.044251 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet aa0aa6cdd0de4b0a91f553c2341e9c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.048386 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet aa0aa6cdd0de4b0a91f553c2341e9c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.055758 2376 ts_tablet_manager.cc:1929] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:38.055886 2376 log.cc:1199] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/a70d70abc3a34dac8cf9a747903fa78c
I20251026 02:56:38.056897 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fc676ef48d7e4c7bbb2fac99cdf17176" } modify_external_catalogs: true
I20251026 02:56:38.057112 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fc676ef48d7e4c7bbb2fac99cdf17176" } modify_external_catalogs: true
I20251026 02:56:38.066180 1661 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 8423becb5b214f32a66445cd351b200f
I20251026 02:56:38.072978 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet de8799a5af2b42cc92ccc0195cff98e7 (table default.Table_8a002df959a4429eb9570d198b363d39 [id=7885bd43f8a245b0a3010a63dbd535f5]). Expected version 2 got 0
I20251026 02:56:38.073699 2377 raft_consensus.cc:2243] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:38.074175 2377 raft_consensus.cc:2272] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.074424 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9973c4eeef944fcb801d4e3d0013d538" } modify_external_catalogs: true
I20251026 02:56:38.074661 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9973c4eeef944fcb801d4e3d0013d538" } modify_external_catalogs: true
I20251026 02:56:38.074873 2377 ts_tablet_manager.cc:1916] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:38.075100 2375 tablet_bootstrap.cc:492] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:38.077307 2375 tablet_bootstrap.cc:654] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:38.078959 2053 consensus_peers.cc:597] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:38.083491 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 863aef239cd34a358a71ddfdba6fb72a consensus state has no leader
I20251026 02:56:38.088436 2372 ts_tablet_manager.cc:1929] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:38.088563 2372 log.cc:1199] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/f6d6f024d8b943d5acf69b64bc7c7214
I20251026 02:56:38.088923 2372 ts_tablet_manager.cc:1950] T f6d6f024d8b943d5acf69b64bc7c7214 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:38.089920 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet de8799a5af2b42cc92ccc0195cff98e7 (table default.Table_8a002df959a4429eb9570d198b363d39 [id=7885bd43f8a245b0a3010a63dbd535f5]). Expected version 2 got 0
I20251026 02:56:38.098631 2376 ts_tablet_manager.cc:1950] T a70d70abc3a34dac8cf9a747903fa78c P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:38.099133 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 524) took 4745 ms. Trace:
I20251026 02:56:38.099231 2376 rpcz_store.cc:276] 1026 02:56:33.353291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.353739 (+ 448us) service_pool.cc:225] Handling call
1026 02:56:37.990544 (+4636805us) ts_tablet_manager.cc:1151] Deleting tablet a70d70abc3a34dac8cf9a747903fa78c
1026 02:56:37.990555 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.052435 (+ 61880us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.055651 (+ 3216us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.099110 (+ 43459us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4636540}
I20251026 02:56:38.099726 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet a70d70abc3a34dac8cf9a747903fa78c (table default.Table_c4bf189953554cb8854613da705a2b42 [id=a84628b0b09f43c69e86344d40641b02]) successfully deleted
W20251026 02:56:38.103796 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 93827.1us
I20251026 02:56:38.104717 2195 leader_election.cc:304] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.104781 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 551) took 5074 ms. Trace:
I20251026 02:56:38.104875 2372 rpcz_store.cc:276] 1026 02:56:33.030451 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.030522 (+ 71us) service_pool.cc:225] Handling call
1026 02:56:37.940534 (+4910012us) ts_tablet_manager.cc:1151] Deleting tablet f6d6f024d8b943d5acf69b64bc7c7214
1026 02:56:37.940545 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.040760 (+100215us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.088312 (+ 47552us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.104758 (+ 16446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4897271}
I20251026 02:56:38.105010 3328 raft_consensus.cc:2804] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:38.105141 3328 raft_consensus.cc:697] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:38.105423 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet f6d6f024d8b943d5acf69b64bc7c7214 (table default.Table_aeec0477dae74164b120683f450c5b94 [id=9fd6e480fc3543149959773e691a3ddd]) successfully deleted
I20251026 02:56:38.105436 3328 consensus_queue.cc:237] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.105659 2372 tablet_replica.cc:333] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:38.106302 2372 raft_consensus.cc:2243] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:38.106631 2372 raft_consensus.cc:2272] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.107132 2372 ts_tablet_manager.cc:1916] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:38.110596 2376 tablet_replica.cc:333] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:38.110873 2376 raft_consensus.cc:2243] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:38.111112 2376 raft_consensus.cc:2272] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.112421 3496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Bootstrap starting.
W20251026 02:56:38.113456 2053 consensus_peers.cc:597] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:38.115231 3372 raft_consensus.cc:493] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.115365 3372 raft_consensus.cc:515] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.115967 3372 leader_election.cc:290] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:38.115993 3239 consensus_queue.cc:1048] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:38.116727 2195 consensus_peers.cc:597] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2524ee0c964345459c3e62f1e39b8f9e. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:38.116986 1916 leader_election.cc:343] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: a08efd37968846d2958e54831488235c
I20251026 02:56:38.117478 2376 ts_tablet_manager.cc:1916] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:38.119066 3006 raft_consensus.cc:493] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.119215 3006 raft_consensus.cc:515] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.119570 2053 leader_election.cc:304] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.119736 3006 leader_election.cc:290] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.119858 3006 raft_consensus.cc:2804] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:38.119948 3006 raft_consensus.cc:697] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:38.120272 3006 consensus_queue.cc:237] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.120600 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44af74033ccf4d048475f12d164938f5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:38.121201 2053 leader_election.cc:343] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.123262 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet e62064fbdffb4954a3a1227efd6327ee (table default.Table_e9da6904da2c4d15bb17df7e9bbb3221 [id=fb1dbbc49ad94d85925be7fc850041e0]) successfully deleted
I20251026 02:56:38.123520 2107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741792cf96164695b03de3fb86b12602" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:38.123690 2107 raft_consensus.cc:3060] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.124403 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "046a34f9408b406ba08471421f422064" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:38.124576 2111 raft_consensus.cc:3060] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.128021 2111 raft_consensus.cc:2468] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:38.130023 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5caeb11322fe48348ebb523967152187" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:38.131676 3510 raft_consensus.cc:493] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.131850 3510 raft_consensus.cc:515] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.132140 2107 raft_consensus.cc:2468] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:38.132472 3510 leader_election.cc:290] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.132365 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be1c0948c0ae4563b7cb644521914cb9" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:38.132759 2118 raft_consensus.cc:3060] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.134316 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9cd10092aac40068a1c7f85e514fd37" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:38.134992 2053 leader_election.cc:343] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.136168 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e2a5e05b70f841cfb0065041b39e89cb" } modify_external_catalogs: true
I20251026 02:56:38.136409 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e2a5e05b70f841cfb0065041b39e89cb" } modify_external_catalogs: true
I20251026 02:56:38.137431 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3050e72d42417ab58a05f7e3aa56b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:38.138168 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet aa0aa6cdd0de4b0a91f553c2341e9c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:38.138883 1915 leader_election.cc:304] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:38.139398 3372 raft_consensus.cc:2749] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:38.141150 1915 leader_election.cc:343] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: a08efd37968846d2958e54831488235c
I20251026 02:56:38.141304 1915 leader_election.cc:304] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:38.141855 3372 raft_consensus.cc:2749] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.135283 2118 raft_consensus.cc:2468] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:38.143352 2195 leader_election.cc:304] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.144758 3517 raft_consensus.cc:2804] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:38.145004 3517 raft_consensus.cc:697] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:38.145309 3517 consensus_queue.cc:237] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:38.151790 2195 consensus_peers.cc:597] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5caeb11322fe48348ebb523967152187. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:38.153923 2195 leader_election.cc:304] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.154407 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8466ce8a41f84c3797f015d455b86d1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.155323 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8466ce8a41f84c3797f015d455b86d1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.155570 3328 raft_consensus.cc:2804] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:38.155781 3328 raft_consensus.cc:697] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:38.156024 3328 consensus_queue.cc:237] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.160655 2376 ts_tablet_manager.cc:1929] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:38.160795 2376 log.cc:1199] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/548a7f9628ac4623b2c2f723f83c1e45
I20251026 02:56:38.161167 2376 ts_tablet_manager.cc:1950] T 548a7f9628ac4623b2c2f723f83c1e45 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:38.161219 2195 leader_election.cc:304] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:38.161526 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 526) took 4800 ms. Trace:
I20251026 02:56:38.161619 2376 rpcz_store.cc:276] 1026 02:56:33.360776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.360833 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:38.110531 (+4749698us) ts_tablet_manager.cc:1151] Deleting tablet 548a7f9628ac4623b2c2f723f83c1e45
1026 02:56:38.110544 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.141738 (+ 31194us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.160524 (+ 18786us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.161503 (+ 979us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4749470}
I20251026 02:56:38.162178 3239 raft_consensus.cc:2804] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:38.162173 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 548a7f9628ac4623b2c2f723f83c1e45 (table default.Table_f8baefeba2b54cb39affb2d4f1b0bef8 [id=d9a6b0b6f3e246a3aa8fbd1167205264]) successfully deleted
I20251026 02:56:38.162309 3239 raft_consensus.cc:697] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:38.162618 3239 consensus_queue.cc:237] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.164615 3510 raft_consensus.cc:493] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.164762 3510 raft_consensus.cc:515] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.165366 3510 leader_election.cc:290] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.166692 2372 ts_tablet_manager.cc:1929] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:38.168805 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e701d4023fee45d18c48997aab3227c2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:38.169996 2372 log.cc:1199] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/903e65fa8c214dc9a50cb97cab101d22
I20251026 02:56:38.170567 2372 ts_tablet_manager.cc:1950] T 903e65fa8c214dc9a50cb97cab101d22 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:38.171895 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 557) took 4808 ms. Trace:
I20251026 02:56:38.173617 2376 tablet_replica.cc:333] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
W20251026 02:56:38.172720 2053 leader_election.cc:343] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.176322 2372 rpcz_store.cc:276] 1026 02:56:33.363142 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.363521 (+ 379us) service_pool.cc:225] Handling call
1026 02:56:38.105607 (+4742086us) ts_tablet_manager.cc:1151] Deleting tablet 903e65fa8c214dc9a50cb97cab101d22
1026 02:56:38.105618 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.137216 (+ 31598us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.166577 (+ 29361us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.171873 (+ 5296us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4741857}
I20251026 02:56:38.176908 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 903e65fa8c214dc9a50cb97cab101d22 (table default.Table_2d168c00cb884360bdb2475bd391f576 [id=65b5770c4b9243b892bfab963a439bb9]) successfully deleted
I20251026 02:56:38.178380 2376 raft_consensus.cc:2243] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:38.180626 2372 tablet_replica.cc:333] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:38.180934 2376 raft_consensus.cc:2272] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.181253 2372 raft_consensus.cc:2243] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
W20251026 02:56:38.181450 2194 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 152859us
I20251026 02:56:38.182014 2376 ts_tablet_manager.cc:1916] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:38.182497 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44af74033ccf4d048475f12d164938f5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:38.182737 2257 raft_consensus.cc:2468] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:38.178156 2378 tablet_bootstrap.cc:492] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:38.183285 2378 ts_tablet_manager.cc:1403] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.166s user 0.006s sys 0.012s
I20251026 02:56:38.184013 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9cd10092aac40068a1c7f85e514fd37" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:38.184288 2378 raft_consensus.cc:359] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.184607 2378 raft_consensus.cc:385] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.184734 2378 raft_consensus.cc:740] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.184988 2378 consensus_queue.cc:260] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.185067 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "046a34f9408b406ba08471421f422064" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:38.185653 2378 ts_tablet_manager.cc:1434] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:38.185910 2378 tablet_bootstrap.cc:492] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:38.187534 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e701d4023fee45d18c48997aab3227c2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:38.187659 2378 tablet_bootstrap.cc:654] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.191020 1974 raft_consensus.cc:3060] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.191926 2372 raft_consensus.cc:2272] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.192548 2372 ts_tablet_manager.cc:1916] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:38.193858 3372 raft_consensus.cc:493] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.194000 3372 raft_consensus.cc:515] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.194626 3372 leader_election.cc:290] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:38.188617 2375 tablet_bootstrap.cc:492] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:38.189363 2053 leader_election.cc:304] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:38.196738 2053 leader_election.cc:343] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.196867 2053 leader_election.cc:304] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:38.197310 2053 leader_election.cc:343] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.197422 2053 leader_election.cc:304] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.197757 3510 raft_consensus.cc:2804] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:38.197824 3510 raft_consensus.cc:493] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:38.197878 3510 raft_consensus.cc:3060] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:38.198239 1916 leader_election.cc:343] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 276bd3b97c97426daf802e1f57feb265
I20251026 02:56:38.200394 3513 raft_consensus.cc:2749] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.200646 3513 raft_consensus.cc:2749] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.201577 2381 tablet_bootstrap.cc:492] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:38.201749 2381 ts_tablet_manager.cc:1403] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.347s user 0.000s sys 0.013s
I20251026 02:56:38.202333 3510 raft_consensus.cc:515] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.203071 3510 leader_election.cc:290] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.203378 3521 raft_consensus.cc:493] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.203517 3521 raft_consensus.cc:515] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.203760 2375 ts_tablet_manager.cc:1403] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.129s user 0.011s sys 0.001s
I20251026 02:56:38.204313 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet fe48472efa444bf580e4074c96a9de92
I20251026 02:56:38.204368 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44af74033ccf4d048475f12d164938f5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:38.204602 2375 raft_consensus.cc:359] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.204749 2375 raft_consensus.cc:385] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.204802 2375 raft_consensus.cc:740] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.205057 3521 leader_election.cc:290] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:38.205013 2375 consensus_queue.cc:260] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:38.205075 2053 leader_election.cc:343] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.205389 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet fe48472efa444bf580e4074c96a9de92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.205643 2375 ts_tablet_manager.cc:1434] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:38.206102 2375 tablet_bootstrap.cc:492] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:38.207090 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet fe48472efa444bf580e4074c96a9de92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.208830 1982 raft_consensus.cc:3060] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.213126 2375 tablet_bootstrap.cc:654] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.206519 3366 raft_consensus.cc:697] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:38.218000 3366 consensus_queue.cc:237] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:38.223587 1915 leader_election.cc:343] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 276bd3b97c97426daf802e1f57feb265
I20251026 02:56:38.223744 1915 leader_election.cc:304] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:38.224246 3372 raft_consensus.cc:2749] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.224738 3516 consensus_queue.cc:1048] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:38.226006 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f3ae3a377af46449c3faca339f86053" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:38.226830 2194 leader_election.cc:343] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:38.245129 2194 consensus_peers.cc:597] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:38.245982 2377 ts_tablet_manager.cc:1929] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251026 02:56:38.246212 2194 leader_election.cc:343] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.249650 2378 tablet_bootstrap.cc:492] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:38.249862 2378 ts_tablet_manager.cc:1403] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.003s
I20251026 02:56:38.250808 2378 raft_consensus.cc:359] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.251077 2378 raft_consensus.cc:385] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.251183 2378 raft_consensus.cc:740] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.251479 2378 consensus_queue.cc:260] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.252127 2378 ts_tablet_manager.cc:1434] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:38.252470 2378 tablet_bootstrap.cc:492] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:38.254231 2378 tablet_bootstrap.cc:654] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.218499 3510 raft_consensus.cc:493] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.256559 3510 raft_consensus.cc:515] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.257300 3510 leader_election.cc:290] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.225175 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f3ae3a377af46449c3faca339f86053" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:38.257495 2117 raft_consensus.cc:2468] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:38.258584 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882fe81da21344ed99ea735fa79f73c0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:38.259436 2053 leader_election.cc:343] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.244390 3006 raft_consensus.cc:493] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.261235 3006 raft_consensus.cc:515] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.266162 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet d90e62ae86b149b291dc29cf7442c6bc
I20251026 02:56:38.266533 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882fe81da21344ed99ea735fa79f73c0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:38.272707 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet d90e62ae86b149b291dc29cf7442c6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.273854 1661 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 63084510d2b745c8b499450e52fd708b
I20251026 02:56:38.274070 3513 consensus_queue.cc:1048] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:38.274210 2053 leader_election.cc:343] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.274305 2053 leader_election.cc:304] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.274694 3272 raft_consensus.cc:2749] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.276558 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet d90e62ae86b149b291dc29cf7442c6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.281134 3006 leader_election.cc:290] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.282600 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_23d0e02f7246407f8337ddd38381fa9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:38.283059 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5b564a01de564f7aa92845f3d962af11" } new_table_name: "default.Table_872292b6a14a4defbc8bcede4cfe81bc" modify_external_catalogs: true
I20251026 02:56:38.283520 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9421cb3b5cd340f5a53162c9855d96b3" } new_table_name: "default.Table_7df0c57231f6477fa64624f1bec15130" modify_external_catalogs: true
I20251026 02:56:38.284639 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_dc184bc582ff43dc9e510f42ebe22815" } new_table_name: "default.Table_f0f7c737d8df4eb09c5408b0b02e954b" modify_external_catalogs: true
I20251026 02:56:38.285959 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b906304b7fc9457f98068c196bdea574" } new_table_name: "default.Table_74bad7faa1614c0fa737ab4293efa31c" modify_external_catalogs: true
I20251026 02:56:38.287340 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_dfc5b9165b6d446380c0e6f7d3752aa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:38.288100 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_653a3f85c24140cdb62a3e9f3ea95fd4" } modify_external_catalogs: true
I20251026 02:56:38.288563 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_12c63af0fb4a44b19e48f3ecb496a2d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:38.288748 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cf4027782c3841c58cea284393344525" } modify_external_catalogs: true
I20251026 02:56:38.288903 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cf4027782c3841c58cea284393344525" } modify_external_catalogs: true
I20251026 02:56:38.289039 2375 tablet_bootstrap.cc:492] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:38.289244 2375 ts_tablet_manager.cc:1403] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.000s
I20251026 02:56:38.289958 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_02d48c3527e64c1e9f77b7a9d89ab372" } new_table_name: "default.Table_8dc3bc43f2194fba9f556ad3a4530cc3" modify_external_catalogs: true
I20251026 02:56:38.289976 2375 raft_consensus.cc:359] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.290174 2375 raft_consensus.cc:385] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.290230 2375 raft_consensus.cc:740] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.290215 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_653a3f85c24140cdb62a3e9f3ea95fd4" } modify_external_catalogs: true
W20251026 02:56:38.291028 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12c63af0fb4a44b19e48f3ecb496a2d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.292855 2376 ts_tablet_manager.cc:1929] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:38.292963 2376 log.cc:1199] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/903e65fa8c214dc9a50cb97cab101d22
I20251026 02:56:38.293303 2376 ts_tablet_manager.cc:1950] T 903e65fa8c214dc9a50cb97cab101d22 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:38.293607 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 525) took 4933 ms. Trace:
I20251026 02:56:38.293671 2376 rpcz_store.cc:276] 1026 02:56:33.360633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.361091 (+ 458us) service_pool.cc:225] Handling call
1026 02:56:38.173559 (+4812468us) ts_tablet_manager.cc:1151] Deleting tablet 903e65fa8c214dc9a50cb97cab101d22
1026 02:56:38.173570 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.215539 (+ 41969us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.292747 (+ 77208us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.293593 (+ 846us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1280,"tablet-delete.queue_time_us":4812315}
I20251026 02:56:38.294155 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 903e65fa8c214dc9a50cb97cab101d22 (table default.Table_2d168c00cb884360bdb2475bd391f576 [id=65b5770c4b9243b892bfab963a439bb9]) successfully deleted
I20251026 02:56:38.294528 3239 consensus_queue.cc:1048] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:38.297199 1974 raft_consensus.cc:3060] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:38.297583 2194 consensus_peers.cc:597] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:38.299275 2195 leader_election.cc:304] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.305110 2372 ts_tablet_manager.cc:1929] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:38.305279 2372 log.cc:1199] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/548a7f9628ac4623b2c2f723f83c1e45
I20251026 02:56:38.305689 2372 ts_tablet_manager.cc:1950] T 548a7f9628ac4623b2c2f723f83c1e45 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:38.306035 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 558) took 4942 ms. Trace:
I20251026 02:56:38.306130 2372 rpcz_store.cc:276] 1026 02:56:33.363235 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.363785 (+ 550us) service_pool.cc:225] Handling call
1026 02:56:38.180562 (+4816777us) ts_tablet_manager.cc:1151] Deleting tablet 548a7f9628ac4623b2c2f723f83c1e45
1026 02:56:38.180575 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.195920 (+ 15345us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.304971 (+109051us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.306017 (+ 1046us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4816619}
I20251026 02:56:38.306625 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 548a7f9628ac4623b2c2f723f83c1e45 (table default.Table_f8baefeba2b54cb39affb2d4f1b0bef8 [id=d9a6b0b6f3e246a3aa8fbd1167205264]) successfully deleted
W20251026 02:56:38.309275 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f197526e7d34f849dbb4f28d8d414cf consensus state has no leader
W20251026 02:56:38.311601 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02861f399f4d488586f279c01eebcdd1 has no consensus state
W20251026 02:56:38.313155 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d7c172802db489788bdc75231aa47d6 consensus state has no leader
I20251026 02:56:38.246325 2377 log.cc:1199] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/03da415f3a944a9a9679b3a45295ce35
I20251026 02:56:38.314074 2377 ts_tablet_manager.cc:1950] T 03da415f3a944a9a9679b3a45295ce35 P 696476265495492790739ec126a08539: Deleting consensus metadata
W20251026 02:56:38.315196 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0dea6ffb68834373a03ced4f0f804473 consensus state has no leader
I20251026 02:56:38.321033 2378 tablet_bootstrap.cc:492] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:38.328045 2375 consensus_queue.cc:260] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.255445 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet fe48472efa444bf580e4074c96a9de92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:38.257292 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44af74033ccf4d048475f12d164938f5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:38.328608 2257 raft_consensus.cc:3060] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.329155 2375 ts_tablet_manager.cc:1434] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20251026 02:56:38.329448 2375 tablet_bootstrap.cc:492] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:38.330977 2378 ts_tablet_manager.cc:1403] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.000s
W20251026 02:56:38.331969 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfc5b9165b6d446380c0e6f7d3752aa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.332392 2257 raft_consensus.cc:2468] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
W20251026 02:56:38.332427 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23d0e02f7246407f8337ddd38381fa9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.333712 3508 tablet.cc:1722] T 1d8cdf79a97149618bcaf0fefee1dfeb P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:38.334686 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 511) took 6931 ms. Trace:
I20251026 02:56:38.334910 2377 rpcz_store.cc:276] 1026 02:56:31.403214 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.404047 (+ 833us) service_pool.cc:225] Handling call
1026 02:56:38.043935 (+6639888us) ts_tablet_manager.cc:1151] Deleting tablet 03da415f3a944a9a9679b3a45295ce35
1026 02:56:38.043946 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.105154 (+ 61208us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.245824 (+140670us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.334631 (+ 88807us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1972,"tablet-delete.queue_time_us":6639684}
I20251026 02:56:38.335573 2377 tablet_replica.cc:333] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:38.335606 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 03da415f3a944a9a9679b3a45295ce35 (table default.Table_828c16a96a47489f835a891f34d707b1 [id=432234d273ca4c9ea53c7a078c274637]) successfully deleted
I20251026 02:56:38.239344 2381 raft_consensus.cc:359] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.314596 3239 raft_consensus.cc:2804] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:38.337829 2381 raft_consensus.cc:385] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.337879 3239 raft_consensus.cc:493] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:38.337975 2381 raft_consensus.cc:740] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.338032 3239 raft_consensus.cc:3060] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.338446 2381 consensus_queue.cc:260] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.339236 2381 ts_tablet_manager.cc:1434] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.137s user 0.001s sys 0.001s
I20251026 02:56:38.339592 2381 tablet_bootstrap.cc:492] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:38.321801 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet d90e62ae86b149b291dc29cf7442c6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:38.341113 2376 tablet_replica.cc:333] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:38.341387 2376 raft_consensus.cc:2243] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:38.341650 2376 raft_consensus.cc:2272] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.341607 3239 raft_consensus.cc:515] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.342409 2372 tablet_replica.cc:333] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:38.342633 2372 raft_consensus.cc:2243] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:38.342845 2372 raft_consensus.cc:2272] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.343075 3239 leader_election.cc:290] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:38.343490 2372 ts_tablet_manager.cc:1916] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:38.346143 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f3ae3a377af46449c3faca339f86053" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:38.346669 2194 leader_election.cc:343] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.345237 2378 raft_consensus.cc:359] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.347678 2378 raft_consensus.cc:385] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.356437 2378 raft_consensus.cc:740] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.356604 2377 raft_consensus.cc:2243] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:38.356873 2378 consensus_queue.cc:260] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.357247 2377 raft_consensus.cc:2272] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.357548 2378 ts_tablet_manager.cc:1434] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20251026 02:56:38.357856 2378 tablet_bootstrap.cc:492] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:38.358166 2377 ts_tablet_manager.cc:1916] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:38.361223 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f3ae3a377af46449c3faca339f86053" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:38.366961 2110 raft_consensus.cc:3060] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.369212 2372 ts_tablet_manager.cc:1929] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:38.369354 2372 log.cc:1199] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/a821ed8a686c452a975acb96b5c74778
I20251026 02:56:38.369721 2372 ts_tablet_manager.cc:1950] T a821ed8a686c452a975acb96b5c74778 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:38.370551 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 560) took 5007 ms. Trace:
I20251026 02:56:38.370656 2372 rpcz_store.cc:276] 1026 02:56:33.363456 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.364455 (+ 999us) service_pool.cc:225] Handling call
1026 02:56:38.306406 (+4941951us) ts_tablet_manager.cc:1151] Deleting tablet a821ed8a686c452a975acb96b5c74778
1026 02:56:38.306415 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.365070 (+ 58655us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.369122 (+ 4052us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.370525 (+ 1403us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4941802}
I20251026 02:56:38.371317 2372 tablet_replica.cc:333] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:38.371523 2372 raft_consensus.cc:2243] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:38.371687 2372 raft_consensus.cc:2272] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.375259 2110 raft_consensus.cc:2468] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:38.375720 2195 leader_election.cc:304] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.376241 3239 raft_consensus.cc:2804] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:38.376407 3239 raft_consensus.cc:697] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:38.376657 3239 consensus_queue.cc:237] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.379769 2375 tablet_bootstrap.cc:654] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.380530 2372 ts_tablet_manager.cc:1916] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:38.392658 3517 raft_consensus.cc:493] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.392866 3517 raft_consensus.cc:515] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.393956 3517 leader_election.cc:290] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:38.394223 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c3050e72d42417ab58a05f7e3aa56b3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:38.394711 2194 leader_election.cc:343] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.406296 3554 tablet.cc:1722] T 1d8cdf79a97149618bcaf0fefee1dfeb P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:38.410945 2378 tablet_bootstrap.cc:654] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.416700 2381 tablet_bootstrap.cc:654] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.427502 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3f3634e9694e4cafab2335c0fab273cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:38.428505 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_465e7833737a45b997d9aa79e8c1bf5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:38.428656 2372 ts_tablet_manager.cc:1929] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:38.428776 2372 log.cc:1199] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/1d7ffdaeec2240dcbe4cafdb96073a7d
I20251026 02:56:38.429136 2372 ts_tablet_manager.cc:1950] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:38.429464 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 562) took 4979 ms. Trace:
I20251026 02:56:38.429538 2372 rpcz_store.cc:276] 1026 02:56:33.450259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.450886 (+ 627us) service_pool.cc:225] Handling call
1026 02:56:38.371264 (+4920378us) ts_tablet_manager.cc:1151] Deleting tablet 1d7ffdaeec2240dcbe4cafdb96073a7d
1026 02:56:38.371276 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.397545 (+ 26269us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.428521 (+ 30976us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.429449 (+ 928us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4920162}
I20251026 02:56:38.429927 2372 tablet_replica.cc:333] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:38.430128 2372 raft_consensus.cc:2243] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:38.426614 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet a821ed8a686c452a975acb96b5c74778 (table default.Table_016812daf097412081aff793ad75f72a [id=347d712aca4e477c89805846800fb824]) successfully deleted
I20251026 02:56:38.430378 2372 pending_rounds.cc:70] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48: Trying to abort 1 pending ops.
I20251026 02:56:38.430424 2372 pending_rounds.cc:77] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888538563084288 op_type: NO_OP noop_request { }
I20251026 02:56:38.430526 2372 raft_consensus.cc:2889] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:38.430604 2372 raft_consensus.cc:2272] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:38.430599 1648 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 59121.6us
I20251026 02:56:38.431028 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 1d7ffdaeec2240dcbe4cafdb96073a7d (table default.Table_7b08384413864da0a28224dd7201c263 [id=a9255e57e54144ba876815cbb6155360]) successfully deleted
I20251026 02:56:38.431173 2372 ts_tablet_manager.cc:1916] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:38.432229 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f3634e9694e4cafab2335c0fab273cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.433122 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c3050e72d42417ab58a05f7e3aa56b3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:38.433300 2117 raft_consensus.cc:2468] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:38.433769 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_465e7833737a45b997d9aa79e8c1bf5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:38.442267 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 168043us
I20251026 02:56:38.443768 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7177931353164b2aa82c00b2807ccd63" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:38.449043 3559 tablet.cc:1722] T 1d8cdf79a97149618bcaf0fefee1dfeb P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:38.451346 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7177931353164b2aa82c00b2807ccd63" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:38.452052 2195 consensus_peers.cc:597] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6d6f024d8b943d5acf69b64bc7c7214. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:38.452811 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet ed836bb7519e4b0290e6547fa57128bc from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:38.453234 2195 leader_election.cc:304] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.453826 3521 raft_consensus.cc:2804] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:38.453979 3521 raft_consensus.cc:493] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:38.454118 3521 raft_consensus.cc:3060] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.454983 2381 tablet_bootstrap.cc:492] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:38.455147 2381 ts_tablet_manager.cc:1403] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.000s
I20251026 02:56:38.455928 2381 raft_consensus.cc:359] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.456118 2381 raft_consensus.cc:385] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.456192 2381 raft_consensus.cc:740] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.456624 2053 leader_election.cc:304] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:38.458225 2053 leader_election.cc:343] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.458689 3547 raft_consensus.cc:2804] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:38.456462 2381 consensus_queue.cc:260] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.459362 3547 raft_consensus.cc:697] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:38.459914 3547 consensus_queue.cc:237] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.460574 3006 raft_consensus.cc:493] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.460589 2381 ts_tablet_manager.cc:1434] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251026 02:56:38.460716 3006 raft_consensus.cc:515] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.461308 3006 leader_election.cc:290] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.461390 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_07a34faac32549c99cb5b5a936a56945"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:38.461699 2381 tablet_bootstrap.cc:492] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:38.462117 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07a34faac32549c99cb5b5a936a56945 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.462630 3006 raft_consensus.cc:493] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.462764 3006 raft_consensus.cc:515] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.463320 3006 leader_election.cc:290] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.469144 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_56c374c6abad4892834544f5df51a5db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.469846 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56c374c6abad4892834544f5df51a5db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.470458 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5f7b25fab37c4983acf922bd78374da6" } modify_external_catalogs: true
I20251026 02:56:38.470676 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5f7b25fab37c4983acf922bd78374da6" } modify_external_catalogs: true
I20251026 02:56:38.471256 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_038ad7af9e93421f9ab149ff87a4c7d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.471654 2053 leader_election.cc:343] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:38.471791 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_038ad7af9e93421f9ab149ff87a4c7d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.471782 2053 leader_election.cc:304] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.472030 3006 raft_consensus.cc:2749] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.472391 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8799a5af2b42cc92ccc0195cff98e7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:38.474556 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 947ad059be4b475f92b5ba208e3dd9e5 consensus state has no leader
I20251026 02:56:38.475203 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ce756cf85c594b7f97cbbfb345be2856"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.475714 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce756cf85c594b7f97cbbfb345be2856 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:38.476104 2053 leader_election.cc:343] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.477027 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8799a5af2b42cc92ccc0195cff98e7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:38.477735 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1ca5b8379c94deab1130bfe8bdc87b0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:38.478461 2053 leader_election.cc:343] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.478577 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1ca5b8379c94deab1130bfe8bdc87b0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:38.478621 2053 leader_election.cc:304] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.478849 3006 raft_consensus.cc:2749] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:38.479278 2053 leader_election.cc:343] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:38.479667 2053 leader_election.cc:343] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.479811 2053 leader_election.cc:304] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.480157 3006 raft_consensus.cc:2749] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.482527 3521 raft_consensus.cc:515] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.483263 3521 leader_election.cc:290] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:38.485598 1982 raft_consensus.cc:3060] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.486299 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c3050e72d42417ab58a05f7e3aa56b3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:38.486486 2114 raft_consensus.cc:3060] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.486932 2375 tablet_bootstrap.cc:492] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:38.487047 2375 ts_tablet_manager.cc:1403] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.158s user 0.002s sys 0.012s
I20251026 02:56:38.487769 2375 raft_consensus.cc:359] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.487924 2375 raft_consensus.cc:385] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.487982 2375 raft_consensus.cc:740] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.488106 1662 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 352a284902c94f088e2e6ae006162db0 (table default.Table_6cd4ee82e9ec406f971669b7f0722677 [id=414375003cd44b5c8c9f0249b339c28a]). Expected version 1 got 0
I20251026 02:56:38.488161 2375 consensus_queue.cc:260] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.488618 2375 ts_tablet_manager.cc:1434] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:38.488883 2375 tablet_bootstrap.cc:492] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:38.489483 2114 raft_consensus.cc:2468] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:38.490044 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c3050e72d42417ab58a05f7e3aa56b3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:38.490836 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 2524ee0c964345459c3e62f1e39b8f9e with delete_type TABLET_DATA_DELETED (replaced by tablet 5a00bc04b3654f90af342986751344e6) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.490904 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 0fe0e06a86e241bc91a1db4219fe25bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:24 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.490904 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 0de06ea2b0544402a979d9ddc487860f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:38.491266 2194 leader_election.cc:343] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.496543 2195 leader_election.cc:304] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.497231 3366 raft_consensus.cc:2804] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:38.497475 3366 raft_consensus.cc:697] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:38.497789 3366 consensus_queue.cc:237] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.490317 2375 tablet_bootstrap.cc:654] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.490031 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet 3ac38e6257e74367b06b15fc49d2af81 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:38.504758 2378 tablet_bootstrap.cc:492] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:38.504998 2378 ts_tablet_manager.cc:1403] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.147s user 0.003s sys 0.009s
I20251026 02:56:38.505825 2378 raft_consensus.cc:359] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.506044 2378 raft_consensus.cc:385] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.506135 2378 raft_consensus.cc:740] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.506402 2378 consensus_queue.cc:260] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.507032 2378 ts_tablet_manager.cc:1434] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:38.507602 2378 tablet_bootstrap.cc:492] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:38.509030 2378 tablet_bootstrap.cc:654] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.511636 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_70e8c0a6af49412e870552049d11eb6c" } modify_external_catalogs: true
I20251026 02:56:38.511863 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_70e8c0a6af49412e870552049d11eb6c" } modify_external_catalogs: true
I20251026 02:56:38.515157 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5a1e2e5e59314659a935626df27c1d18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.516256 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a1e2e5e59314659a935626df27c1d18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.515700 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_72ee520a0e4d45fb920d1e3cc7a426b3" } modify_external_catalogs: true
I20251026 02:56:38.517583 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_72ee520a0e4d45fb920d1e3cc7a426b3" } modify_external_catalogs: true
I20251026 02:56:38.522096 2378 tablet_bootstrap.cc:492] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:38.522253 2378 ts_tablet_manager.cc:1403] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20251026 02:56:38.523981 2377 ts_tablet_manager.cc:1929] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:38.524730 2377 log.cc:1199] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/86d8e1e21c5f440d8baf76c357d01ad2
W20251026 02:56:38.525086 1647 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2f197526e7d34f849dbb4f28d8d414cf (table default.Table_74bad7faa1614c0fa737ab4293efa31c [id=fa7ef54b9b634a779bd5c6d450843d5f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2f197526e7d34f849dbb4f28d8d414cf consensus state has no leader
I20251026 02:56:38.525445 2377 ts_tablet_manager.cc:1950] T 86d8e1e21c5f440d8baf76c357d01ad2 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:38.526458 2378 raft_consensus.cc:359] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.527436 2378 raft_consensus.cc:385] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.527503 2378 raft_consensus.cc:740] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.527705 2378 consensus_queue.cc:260] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.529613 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 512) took 7122 ms. Trace:
I20251026 02:56:38.529778 2377 rpcz_store.cc:276] 1026 02:56:31.403399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.404293 (+ 894us) service_pool.cc:225] Handling call
1026 02:56:38.335494 (+6931201us) ts_tablet_manager.cc:1151] Deleting tablet 86d8e1e21c5f440d8baf76c357d01ad2
1026 02:56:38.335505 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.480511 (+145006us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.515646 (+ 35135us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.526028 (+ 10382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6930990}
I20251026 02:56:38.530310 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4996540a4e3749c6aa9bdecf10b2fae3" } new_table_name: "default.Table_67c88a3f6e1a48a096b9c860793617ba" modify_external_catalogs: true
I20251026 02:56:38.530632 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 86d8e1e21c5f440d8baf76c357d01ad2 (table default.Table_689f802c9e4b457eb71fb78898fe14b0 [id=05242557de9a4a19948b8591ce73ab28]) successfully deleted
I20251026 02:56:38.530906 2381 tablet_bootstrap.cc:654] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.532706 2378 ts_tablet_manager.cc:1434] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251026 02:56:38.535434 2375 tablet_bootstrap.cc:492] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:38.535672 2375 ts_tablet_manager.cc:1403] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.005s
I20251026 02:56:38.536720 2375 raft_consensus.cc:359] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.537025 2375 raft_consensus.cc:385] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.537137 2375 raft_consensus.cc:740] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.537434 2375 consensus_queue.cc:260] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:38.539436 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6cbea840b88c4ef6b140a9a769fd69df consensus state has no leader
I20251026 02:56:38.540285 2372 ts_tablet_manager.cc:1929] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:38.540422 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7ea22608d74d41f4853bb29edc7e5c52" } new_table_name: "default.Table_17551d2211c14255842b226a8b8fcdd1" modify_external_catalogs: true
I20251026 02:56:38.549736 2372 log.cc:1199] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/cfc3aed64ad0439cb22fd6815c973d87
W20251026 02:56:38.550107 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1406c22ee4ee4f94acecad9546208e79 consensus state has no leader
I20251026 02:56:38.544039 2375 ts_tablet_manager.cc:1434] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251026 02:56:38.550521 2375 tablet_bootstrap.cc:492] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:38.552462 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 513) took 7131 ms. Trace:
I20251026 02:56:38.552609 2377 rpcz_store.cc:276] 1026 02:56:31.405293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.405993 (+ 700us) service_pool.cc:225] Handling call
1026 02:56:38.536257 (+7130264us) ts_tablet_manager.cc:1151] Deleting tablet 8dbc12d49b094dbeadad2e76916ba9af
1026 02:56:38.536268 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.536405 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7130048}
I20251026 02:56:38.552650 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0dfae834c6294764ae2c58c6a11c5280" } new_table_name: "default.Table_1faa0c8151724c5e816546ee2e1d5182" modify_external_catalogs: true
W20251026 02:56:38.553063 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 8dbc12d49b094dbeadad2e76916ba9af because the tablet was not found. No further retry: Not found: Tablet not found: 8dbc12d49b094dbeadad2e76916ba9af
I20251026 02:56:38.556180 2377 tablet_replica.cc:333] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:38.556468 2377 raft_consensus.cc:2243] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:38.556883 2377 raft_consensus.cc:2272] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.559369 2377 ts_tablet_manager.cc:1916] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:38.566105 2378 tablet_bootstrap.cc:492] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:38.567507 2381 tablet_bootstrap.cc:492] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:38.567754 2381 ts_tablet_manager.cc:1403] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.001s
I20251026 02:56:38.568638 2372 ts_tablet_manager.cc:1950] T cfc3aed64ad0439cb22fd6815c973d87 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:38.569550 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 563) took 5119 ms. Trace:
I20251026 02:56:38.569660 2372 rpcz_store.cc:276] 1026 02:56:33.450426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.451141 (+ 715us) service_pool.cc:225] Handling call
1026 02:56:38.429875 (+4978734us) ts_tablet_manager.cc:1151] Deleting tablet cfc3aed64ad0439cb22fd6815c973d87
1026 02:56:38.429887 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.501474 (+ 71587us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.540148 (+ 38674us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.569526 (+ 29378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4978583}
I20251026 02:56:38.570298 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet cfc3aed64ad0439cb22fd6815c973d87 (table default.Table_34881d9847d5463297830bf6cc4ed1e6 [id=a484f36091f24fd2b8bc2af4f53eb450]) successfully deleted
I20251026 02:56:38.570500 2372 tablet_replica.cc:333] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:38.570820 2372 raft_consensus.cc:2243] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:38.571079 2372 raft_consensus.cc:2272] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.571935 2372 ts_tablet_manager.cc:1916] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:38.572125 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41a1442fffe54add93666e3c78a0cc58 consensus state has no leader
I20251026 02:56:38.573067 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a37eaaa2d8244f6f95a762d240e1dffb" } new_table_name: "default.Table_0982fce6dbaa41f69b6e80e254f245db" modify_external_catalogs: true
I20251026 02:56:38.576354 2375 tablet_bootstrap.cc:654] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.581243 2378 tablet_bootstrap.cc:654] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:38.588299 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 885c10070d6942f7abe52c4558d887e6 consensus state has no leader
I20251026 02:56:38.594178 3006 raft_consensus.cc:493] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.594426 3006 raft_consensus.cc:515] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.595100 3006 leader_election.cc:290] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.595724 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa10cd990a944eab8d9eb66aaf20080c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:38.596297 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa10cd990a944eab8d9eb66aaf20080c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:38.596858 2053 leader_election.cc:343] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:38.597158 2053 leader_election.cc:343] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.597296 2053 leader_election.cc:304] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.597676 3006 raft_consensus.cc:2749] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.598407 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 6a3491b303f94fe2a2c838297f73d464 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:38.599969 2375 tablet_bootstrap.cc:492] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:38.600194 2375 ts_tablet_manager.cc:1403] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.004s
I20251026 02:56:38.601060 2375 raft_consensus.cc:359] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.601373 2375 raft_consensus.cc:385] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.601480 2375 raft_consensus.cc:740] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.601853 2375 consensus_queue.cc:260] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.602874 2375 ts_tablet_manager.cc:1434] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251026 02:56:38.611377 2381 raft_consensus.cc:359] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.612118 2375 tablet_bootstrap.cc:492] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:38.611600 2381 raft_consensus.cc:385] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.613798 2381 raft_consensus.cc:740] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.614030 2381 consensus_queue.cc:260] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.615893 2378 tablet_bootstrap.cc:492] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:38.616039 2378 ts_tablet_manager.cc:1403] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20251026 02:56:38.616847 2378 raft_consensus.cc:359] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.617013 2378 raft_consensus.cc:385] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.617064 2378 raft_consensus.cc:740] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.617287 2378 consensus_queue.cc:260] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.617681 1657 catalog_manager.cc:5649] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:38.618187 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 046a34f9408b406ba08471421f422064 (table default.Table_760b799491d7426a86aa1dc3c76fab31 [id=bc8a2d95ed134fae9f3e9deeac4f130a]). Expected version 3 got 0
I20251026 02:56:38.619534 2378 ts_tablet_manager.cc:1434] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251026 02:56:38.619812 2378 tablet_bootstrap.cc:492] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:38.621413 2378 tablet_bootstrap.cc:654] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.623968 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (table default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5]). Expected version 1 got 0
I20251026 02:56:38.624166 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (table default.Table_1fa437ef7f89450bb6f672a1fb491baa [id=44c37945ff9f48e8b2c07648394ac177]). Expected version 1 got 0
I20251026 02:56:38.624416 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 2fd8f02126b24285bbfa575540ba0f09 (table default.Table_7539d36a088f4eafaa21d151ed9ea1cb [id=1950d7cb826b4a99b1ba8866e1d845f3]). Expected version 1 got 0
I20251026 02:56:38.628895 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0cf76ef8f6af445eb0ce15c8d79136c5
I20251026 02:56:38.628961 3372 raft_consensus.cc:493] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.629074 3372 raft_consensus.cc:515] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.629698 3372 leader_election.cc:290] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:38.630155 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cf76ef8f6af445eb0ce15c8d79136c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.630337 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cf76ef8f6af445eb0ce15c8d79136c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.631153 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cf76ef8f6af445eb0ce15c8d79136c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:38.631757 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e9b0e1d3f3f4fdbb8c58f26a4308771" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:38.632781 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e9b0e1d3f3f4fdbb8c58f26a4308771" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:38.633365 1915 leader_election.cc:304] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:38.633574 2377 ts_tablet_manager.cc:1929] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:38.633677 2377 log.cc:1199] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/0efdcc36ff5b47379e0a76512e05d379
I20251026 02:56:38.633862 3372 raft_consensus.cc:2749] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.634044 2377 ts_tablet_manager.cc:1950] T 0efdcc36ff5b47379e0a76512e05d379 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:38.634383 2378 tablet_bootstrap.cc:492] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:38.634421 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 524) took 6965 ms. Trace:
I20251026 02:56:38.634631 2378 ts_tablet_manager.cc:1403] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20251026 02:56:38.634522 2377 rpcz_store.cc:276] 1026 02:56:31.669247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.669306 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:38.553037 (+6883731us) ts_tablet_manager.cc:1151] Deleting tablet 0efdcc36ff5b47379e0a76512e05d379
1026 02:56:38.556062 (+ 3025us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.596882 (+ 40820us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.633451 (+ 36569us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.634404 (+ 953us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6883429}
I20251026 02:56:38.634956 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 526) took 6852 ms. Trace:
I20251026 02:56:38.635028 2377 rpcz_store.cc:276] 1026 02:56:31.782289 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.782359 (+ 70us) service_pool.cc:225] Handling call
1026 02:56:38.634827 (+6852468us) ts_tablet_manager.cc:1151] Deleting tablet c31d2704d1f14ddead69c794cc631ec1
1026 02:56:38.634835 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.634944 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6852196}
I20251026 02:56:38.635290 2377 tablet_replica.cc:333] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:38.635238 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 0efdcc36ff5b47379e0a76512e05d379 (table default.Table_1972d81a4c494b7c97894d4991fed574 [id=d9f8b0fbb3ed416fb380a5717fb29dba]) successfully deleted
I20251026 02:56:38.635464 2377 raft_consensus.cc:2243] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:38.635800 2377 raft_consensus.cc:2272] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.636554 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 1dec51b88e3545d1a0a7793613540ee3 (DEFAULT_TABLE table=default.Table_e6acd68207564173ac53389222b0c621 [id=c6ea9ee3b1a749609a08e993826b74e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.636943 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dec51b88e3545d1a0a7793613540ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.614542 2381 ts_tablet_manager.cc:1434] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251026 02:56:38.637959 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 2f1872ef9f244a388d25a8085dd53f82
I20251026 02:56:38.638979 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f1872ef9f244a388d25a8085dd53f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.639894 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f1872ef9f244a388d25a8085dd53f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.640790 2375 tablet_bootstrap.cc:654] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.643572 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3ac38e6257e74367b06b15fc49d2af81
I20251026 02:56:38.644073 1662 catalog_manager.cc:6719] ReplaceTablet: tablet 3ac38e6257e74367b06b15fc49d2af81 deleted and replaced by tablet 7c148c74fbcd4d30b9ba0ea059284427
I20251026 02:56:38.644726 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ac38e6257e74367b06b15fc49d2af81 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c148c74fbcd4d30b9ba0ea059284427) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.645081 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 62d30d79a6e4432a8dbe40a104e87fa0
I20251026 02:56:38.645920 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 5c6e3e47b3e7421e945cfbe671b06abe
I20251026 02:56:38.646672 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 62d30d79a6e4432a8dbe40a104e87fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.647328 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c6e3e47b3e7421e945cfbe671b06abe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.648859 2378 raft_consensus.cc:359] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.649053 2378 raft_consensus.cc:385] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.649109 2378 raft_consensus.cc:740] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.649351 2378 consensus_queue.cc:260] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.650219 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 62d30d79a6e4432a8dbe40a104e87fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.650219 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c6e3e47b3e7421e945cfbe671b06abe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.650489 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 863aef239cd34a358a71ddfdba6fb72a
I20251026 02:56:38.651477 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 863aef239cd34a358a71ddfdba6fb72a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.655290 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ac38e6257e74367b06b15fc49d2af81 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c148c74fbcd4d30b9ba0ea059284427) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.655812 2378 ts_tablet_manager.cc:1434] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20251026 02:56:38.657718 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e9c6f4d328654c33b5edf5cf73450028"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.658377 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9c6f4d328654c33b5edf5cf73450028 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.659543 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (table default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5]). Expected version 1 got 0
I20251026 02:56:38.659832 1659 catalog_manager.cc:5649] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:38.660375 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 8f3ae3a377af46449c3faca339f86053 (table default.Table_42e3536941484e83a992e13086dc316e [id=31f977e55ba041f59bcc721d469746fd]). Expected version 2 got 0
I20251026 02:56:38.660703 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet dba102838945453cbf44e1cfe505a52c (table default.Table_8ffce6cf6b2d44aa8a9459e834de1d80 [id=92310ee99b5c428c9773b0f7a5462a4d]). Expected version 1 got 0
I20251026 02:56:38.661132 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (table default.Table_1fa437ef7f89450bb6f672a1fb491baa [id=44c37945ff9f48e8b2c07648394ac177]). Expected version 1 got 0
I20251026 02:56:38.661381 1655 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_66d05ce30ff642b58cc2d6658d509575 [id=690ec15818d34cdfa990c8fcd36a19cd] alter complete (version 2)
I20251026 02:56:38.663450 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 7177931353164b2aa82c00b2807ccd63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.640288 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f1872ef9f244a388d25a8085dd53f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:38.637701 2381 tablet_bootstrap.cc:492] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:38.641182 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 86d8e1e21c5f440d8baf76c357d01ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:38.650863 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ac38e6257e74367b06b15fc49d2af81 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c148c74fbcd4d30b9ba0ea059284427) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:38.650954 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 62d30d79a6e4432a8dbe40a104e87fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:38.651052 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c6e3e47b3e7421e945cfbe671b06abe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:38.659487 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet 863aef239cd34a358a71ddfdba6fb72a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:38.667160 2377 ts_tablet_manager.cc:1916] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:38.667806 2027 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:38.674723 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_922fb2f689694180b885eb3a1528cb24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.675350 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_922fb2f689694180b885eb3a1528cb24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.675925 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 863aef239cd34a358a71ddfdba6fb72a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.676316 2378 tablet_bootstrap.cc:492] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:38.676616 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_35945e2ea4504c3c8db41203934516df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.677111 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35945e2ea4504c3c8db41203934516df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.677748 2378 tablet_bootstrap.cc:654] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.679157 2372 ts_tablet_manager.cc:1929] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:38.679452 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fb45779e61d94a69ab237d499fdaea52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.679970 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb45779e61d94a69ab237d499fdaea52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.680785 2372 log.cc:1199] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/efd751ee4b1d433198dcd00c081f648d
I20251026 02:56:38.681293 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8a002df959a4429eb9570d198b363d39" } new_table_name: "default.Table_13667e60be9c46b192cebf20df523e99" modify_external_catalogs: true
I20251026 02:56:38.684406 2372 ts_tablet_manager.cc:1950] T efd751ee4b1d433198dcd00c081f648d P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:38.685827 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (table default.Table_1fa437ef7f89450bb6f672a1fb491baa [id=44c37945ff9f48e8b2c07648394ac177]). Expected version 1 got 0
I20251026 02:56:38.687258 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e20051ad208b42e39e779968848a4c86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:38.687671 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet f437d8b5f5784e8fbec10d0700e90348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:38.687952 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e20051ad208b42e39e779968848a4c86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.688534 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet aa10cd990a944eab8d9eb66aaf20080c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.689695 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (table default.Table_1fa437ef7f89450bb6f672a1fb491baa [id=44c37945ff9f48e8b2c07648394ac177]). Expected version 1 got 0
I20251026 02:56:38.689944 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (table default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5]). Expected version 1 got 0
I20251026 02:56:38.693452 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 564) took 5242 ms. Trace:
I20251026 02:56:38.695312 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet efd751ee4b1d433198dcd00c081f648d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:38.699841 2376 ts_tablet_manager.cc:1916] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:38.699256 2372 rpcz_store.cc:276] 1026 02:56:33.450579 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.451340 (+ 761us) service_pool.cc:225] Handling call
1026 02:56:38.570355 (+5119015us) ts_tablet_manager.cc:1151] Deleting tablet efd751ee4b1d433198dcd00c081f648d
1026 02:56:38.570368 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.643877 (+ 73509us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.674856 (+ 30979us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.693426 (+ 18570us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5118855}
I20251026 02:56:38.702533 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet efd751ee4b1d433198dcd00c081f648d (table default.Table_c2ef795e86444f9f85bb7d0bcca51240 [id=20d9432d9f6443099d7fc0acb952a839]) successfully deleted
I20251026 02:56:38.704268 2372 tablet_replica.cc:333] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:38.707451 3006 raft_consensus.cc:493] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.707599 3006 raft_consensus.cc:515] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.708246 3006 leader_election.cc:290] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.706821 2372 raft_consensus.cc:2243] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
W20251026 02:56:38.709123 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de8799a5af2b42cc92ccc0195cff98e7 consensus state has no leader
I20251026 02:56:38.709636 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3d11f056c2674f46a40d4fe0e8473326"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:38.709761 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d212170f056a44dfb15cfcb7f9fbae2f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:38.710278 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d212170f056a44dfb15cfcb7f9fbae2f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:38.710618 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d11f056c2674f46a40d4fe0e8473326 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:38.710913 2053 leader_election.cc:343] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:38.711410 2053 leader_election.cc:343] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.711529 2053 leader_election.cc:304] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.712252 3510 raft_consensus.cc:2749] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.712919 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2fab98cc17b5470a934ece6a4120a664"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.713505 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fab98cc17b5470a934ece6a4120a664 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.714352 2377 ts_tablet_manager.cc:1929] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:38.715317 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_cbd8bd7117a34f5a836578a05d997586"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.715848 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbd8bd7117a34f5a836578a05d997586 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.720091 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 1dec51b88e3545d1a0a7793613540ee3 (DEFAULT_TABLE table=default.Table_e6acd68207564173ac53389222b0c621 [id=c6ea9ee3b1a749609a08e993826b74e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.720289 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 905541bcc7644c36976cc987dc6a1d3c (DEFAULT_TABLE table=default.Table_95c013f9c71247aeb254654850f8be47 [id=0d6da4b71f4045848df9edd4a684e057]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.720571 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dec51b88e3545d1a0a7793613540ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.720698 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 905541bcc7644c36976cc987dc6a1d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.725606 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet cfc3aed64ad0439cb22fd6815c973d87 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:38.727254 1943 tablet_service.cc:1505] Processing CreateTablet for tablet c0e6905e5ac147e3bc3a0073816faf76 (DEFAULT_TABLE table=default.Table_e3e8bac83d124aacb292bdec63fbb5fb [id=4820a040872241fcb74f10b9dcb4dde3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.727738 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e6905e5ac147e3bc3a0073816faf76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.728338 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 905541bcc7644c36976cc987dc6a1d3c (DEFAULT_TABLE table=default.Table_95c013f9c71247aeb254654850f8be47 [id=0d6da4b71f4045848df9edd4a684e057]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.728734 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 905541bcc7644c36976cc987dc6a1d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.732172 1962 tablet_service.cc:1505] Processing CreateTablet for tablet a8e8959e8af84742beb00e3ea2f2f4c0 (DEFAULT_TABLE table=default.Table_e74211bb29f145a8aeec9c6f0dab4bff [id=2ffcdca25c58488c849387f50016c42d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.732456 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 4568b52fb082412490aa5fe8da4fdc35 (DEFAULT_TABLE table=default.Table_ed035c4e56264e5c8401998633ac2a04 [id=54273ff4359f42348569f36ff845ca6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.732590 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e8959e8af84742beb00e3ea2f2f4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.732733 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4568b52fb082412490aa5fe8da4fdc35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.735801 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_40efa920a1f94812ab827515fddc0033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.736533 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40efa920a1f94812ab827515fddc0033 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:38.738075 3510 raft_consensus.cc:493] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.738231 3510 raft_consensus.cc:515] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.738309 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 4568b52fb082412490aa5fe8da4fdc35 (DEFAULT_TABLE table=default.Table_ed035c4e56264e5c8401998633ac2a04 [id=54273ff4359f42348569f36ff845ca6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.738673 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4568b52fb082412490aa5fe8da4fdc35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.738823 3510 leader_election.cc:290] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.739207 3372 raft_consensus.cc:493] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.739338 3372 raft_consensus.cc:515] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.739938 3372 leader_election.cc:290] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:38.740756 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f2e34abc2f4148f390dd8ccfda7e4fb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:38.741333 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2e34abc2f4148f390dd8ccfda7e4fb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:38.741505 1916 leader_election.cc:343] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 0fe0e06a86e241bc91a1db4219fe25bc
I20251026 02:56:38.741534 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_461dd114dc774ff7bb820f85a2acc574" } new_table_name: "default.Table_a92e68d05f4b455d810180ee4cc5874d" modify_external_catalogs: true
I20251026 02:56:38.738087 2099 tablet_service.cc:1505] Processing CreateTablet for tablet c0e6905e5ac147e3bc3a0073816faf76 (DEFAULT_TABLE table=default.Table_e3e8bac83d124aacb292bdec63fbb5fb [id=4820a040872241fcb74f10b9dcb4dde3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.742026 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e6905e5ac147e3bc3a0073816faf76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.745734 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8688a0bf946841deb5d88c931538b129" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:38.746325 2053 leader_election.cc:343] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.746331 2089 tablet_service.cc:1505] Processing CreateTablet for tablet a8e8959e8af84742beb00e3ea2f2f4c0 (DEFAULT_TABLE table=default.Table_e74211bb29f145a8aeec9c6f0dab4bff [id=2ffcdca25c58488c849387f50016c42d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.746719 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e8959e8af84742beb00e3ea2f2f4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.748414 3372 raft_consensus.cc:493] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:38.748550 3372 raft_consensus.cc:515] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.749174 3372 leader_election.cc:290] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:38.749838 1629 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 73389.2us
W20251026 02:56:38.750706 1916 leader_election.cc:343] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 115ccccf16b74efc9f24b607cf0c74d1
W20251026 02:56:38.751049 1915 leader_election.cc:343] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 115ccccf16b74efc9f24b607cf0c74d1
I20251026 02:56:38.751149 1915 leader_election.cc:304] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:38.752167 3372 raft_consensus.cc:2749] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:38.753942 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_aeec0477dae74164b120683f450c5b94" } new_table_name: "default.Table_30b96d2d4369458390803c6be18efe0e" modify_external_catalogs: true
I20251026 02:56:38.755728 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 18244445760a434ba7ab2f3a974da7a0 (DEFAULT_TABLE table=default.Table_dc22b2aa5fe64641b34fa498992594cc [id=f4513ca6bc3b49d9a8e985d821988ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.756644 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 18244445760a434ba7ab2f3a974da7a0 (DEFAULT_TABLE table=default.Table_dc22b2aa5fe64641b34fa498992594cc [id=f4513ca6bc3b49d9a8e985d821988ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.756202 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18244445760a434ba7ab2f3a974da7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.757160 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18244445760a434ba7ab2f3a974da7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.759330 2378 tablet_bootstrap.cc:492] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:38.759528 2378 ts_tablet_manager.cc:1403] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.083s user 0.013s sys 0.001s
I20251026 02:56:38.760257 2378 raft_consensus.cc:359] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.764302 2378 raft_consensus.cc:385] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.764453 2378 raft_consensus.cc:740] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.771701 2375 tablet_bootstrap.cc:492] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:38.761018 2376 ts_tablet_manager.cc:1929] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:38.765642 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 02861f399f4d488586f279c01eebcdd1 (DEFAULT_TABLE table=default.Table_f0f7c737d8df4eb09c5408b0b02e954b [id=93c4ab50163c427b8cec6855adb3df48]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:38.772464 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b562b7b51f94b268c667650df199fbd consensus state has no leader
I20251026 02:56:38.772578 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02861f399f4d488586f279c01eebcdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.771962 2375 ts_tablet_manager.cc:1403] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.160s user 0.006s sys 0.004s
W20251026 02:56:38.773223 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 398c1f420974453b9b6b0fe17795378b consensus state has no leader
I20251026 02:56:38.773882 2375 raft_consensus.cc:359] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.774164 2375 raft_consensus.cc:385] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.774292 2375 raft_consensus.cc:740] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.774601 2375 consensus_queue.cc:260] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.776330 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2ec6f8b4f5a1475c91edabb9dbceced9" } new_table_name: "default.Table_4d51b54e2a28409fac865e70f2444c78" modify_external_catalogs: true
I20251026 02:56:38.780320 2376 log.cc:1199] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/a821ed8a686c452a975acb96b5c74778
I20251026 02:56:38.780298 3006 raft_consensus.cc:493] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.780442 3006 raft_consensus.cc:515] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.780711 2376 ts_tablet_manager.cc:1950] T a821ed8a686c452a975acb96b5c74778 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:38.781080 3006 leader_election.cc:290] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.781513 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 527) took 5418 ms. Trace:
I20251026 02:56:38.781597 2376 rpcz_store.cc:276] 1026 02:56:33.362768 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.362818 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:38.341053 (+4978235us) ts_tablet_manager.cc:1151] Deleting tablet a821ed8a686c452a975acb96b5c74778
1026 02:56:38.341066 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.724330 (+383264us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.760928 (+ 36598us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.781488 (+ 20560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4978034}
I20251026 02:56:38.781792 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe48472efa444bf580e4074c96a9de92" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:38.781967 2376 tablet_replica.cc:333] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:38.782202 2378 consensus_queue.cc:260] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:38.782372 2053 leader_election.cc:343] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.782989 2378 ts_tablet_manager.cc:1434] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s
I20251026 02:56:38.783264 2378 tablet_bootstrap.cc:492] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:38.784519 2375 ts_tablet_manager.cc:1434] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20251026 02:56:38.785017 2378 tablet_bootstrap.cc:654] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.786561 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 02861f399f4d488586f279c01eebcdd1 (DEFAULT_TABLE table=default.Table_f0f7c737d8df4eb09c5408b0b02e954b [id=93c4ab50163c427b8cec6855adb3df48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.787045 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02861f399f4d488586f279c01eebcdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.787340 2375 tablet_bootstrap.cc:492] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:38.792836 2372 pending_rounds.cc:70] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48: Trying to abort 1 pending ops.
I20251026 02:56:38.792972 2372 pending_rounds.cc:77] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888542106931200 op_type: NO_OP noop_request { }
I20251026 02:56:38.793165 2372 raft_consensus.cc:2889] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:38.793339 2372 raft_consensus.cc:2272] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.764164 3496 log.cc:826] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Log is configured to *not* fsync() on all Append() calls
I20251026 02:56:38.795730 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet a821ed8a686c452a975acb96b5c74778 (table default.Table_016812daf097412081aff793ad75f72a [id=347d712aca4e477c89805846800fb824]) successfully deleted
I20251026 02:56:38.795871 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a463861b69f34f099edf69c1eb7d99a6" } new_table_name: "default.Table_548905e23d3c442c928134d3e7a2bb03" modify_external_catalogs: true
I20251026 02:56:38.796957 2099 tablet_service.cc:1505] Processing CreateTablet for tablet df585f11274f4fd2852890aa2c53f0c0 (DEFAULT_TABLE table=default.Table_a463861b69f34f099edf69c1eb7d99a6 [id=58af1b534f644c758e06ca97aa86fce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.797869 2089 tablet_service.cc:1505] Processing CreateTablet for tablet bfd54c88330e4ebba6c07fca53cacc53 (DEFAULT_TABLE table=default.Table_f25a397301e7496a8d70b9a34ae32f6b [id=48e820dac44640fdaa74eb0c1ef4d0e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.798388 2096 tablet_service.cc:1505] Processing CreateTablet for tablet ade0f7de597049ffb45274d60c9c0932 (DEFAULT_TABLE table=default.Table_b67009abd92345148b7cb2d1be01e4a9 [id=de5ae8e655474ee7b05f3ed272288251]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.798563 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df585f11274f4fd2852890aa2c53f0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.791787 3372 raft_consensus.cc:493] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.802073 3372 raft_consensus.cc:515] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.802706 3372 leader_election.cc:290] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:38.803607 1916 leader_election.cc:343] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 2524ee0c964345459c3e62f1e39b8f9e
I20251026 02:56:38.805830 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade0f7de597049ffb45274d60c9c0932. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.807337 2094 tablet_service.cc:1505] Processing CreateTablet for tablet a43c7d16330644b5bfa0a88e5b624f6b (DEFAULT_TABLE table=default.Table_20aea86003f649d282631fee5df071c8 [id=8865de0dc99d477f8adc5ae8a3a502ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.807698 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43c7d16330644b5bfa0a88e5b624f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.808516 1943 tablet_service.cc:1505] Processing CreateTablet for tablet bfd54c88330e4ebba6c07fca53cacc53 (DEFAULT_TABLE table=default.Table_f25a397301e7496a8d70b9a34ae32f6b [id=48e820dac44640fdaa74eb0c1ef4d0e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.808863 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfd54c88330e4ebba6c07fca53cacc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.815706 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfd54c88330e4ebba6c07fca53cacc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.816879 1947 tablet_service.cc:1505] Processing CreateTablet for tablet 0aabfe11dedd454f8221fb5de25a268b (DEFAULT_TABLE table=default.Table_6fc619ce52194bd58287fde601a7a24d [id=1b8eca019058481ebe80d9cb89ed1590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.817307 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aabfe11dedd454f8221fb5de25a268b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:38.817565 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 109073eb664d4090b7a2b786146e13fa consensus state has no leader
I20251026 02:56:38.818815 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 0aabfe11dedd454f8221fb5de25a268b (DEFAULT_TABLE table=default.Table_6fc619ce52194bd58287fde601a7a24d [id=1b8eca019058481ebe80d9cb89ed1590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.819139 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aabfe11dedd454f8221fb5de25a268b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:38.820869 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df585f11274f4fd2852890aa2c53f0c0 consensus state has no leader
I20251026 02:56:38.822747 2372 ts_tablet_manager.cc:1916] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:38.823930 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 0b1f0e8f18c84576a574efd9f46ad458 (DEFAULT_TABLE table=default.Table_09e15e381f1d41dca2fb4079688213db [id=9b6cbcd7ee0c48e0b1d21a00371046b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.824349 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b1f0e8f18c84576a574efd9f46ad458. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.825287 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 0b1f0e8f18c84576a574efd9f46ad458 (DEFAULT_TABLE table=default.Table_09e15e381f1d41dca2fb4079688213db [id=9b6cbcd7ee0c48e0b1d21a00371046b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.825623 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b1f0e8f18c84576a574efd9f46ad458. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.826514 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 97aa7f0266854c0a8f17e6bceac0c9b5 (DEFAULT_TABLE table=default.Table_50ea87134e5c4bba934f9f7a5e275c9b [id=34a9fbe2da4d47e9a600083ae1a85971]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.826808 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97aa7f0266854c0a8f17e6bceac0c9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.830806 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 992a43f2dabf4e13865e4c128aae763e (DEFAULT_TABLE table=default.Table_6b7d3c3d1bbc40f9b1ef1a34f2f5332f [id=75a2e41b0d134dcdb1c0bc3ea474c467]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.831197 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 992a43f2dabf4e13865e4c128aae763e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.833550 1982 raft_consensus.cc:3060] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.834404 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 992a43f2dabf4e13865e4c128aae763e (DEFAULT_TABLE table=default.Table_6b7d3c3d1bbc40f9b1ef1a34f2f5332f [id=75a2e41b0d134dcdb1c0bc3ea474c467]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.834777 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 992a43f2dabf4e13865e4c128aae763e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.836997 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a22a7510b6a249398d85c3fac300f90a" } modify_external_catalogs: true
I20251026 02:56:38.837131 2375 tablet_bootstrap.cc:654] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.837181 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a22a7510b6a249398d85c3fac300f90a" } modify_external_catalogs: true
I20251026 02:56:38.838253 1955 tablet_service.cc:1505] Processing CreateTablet for tablet df585f11274f4fd2852890aa2c53f0c0 (DEFAULT_TABLE table=default.Table_a463861b69f34f099edf69c1eb7d99a6 [id=58af1b534f644c758e06ca97aa86fce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.838598 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df585f11274f4fd2852890aa2c53f0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.841578 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7ece4924086a4ae68db59650193f0e0e" } modify_external_catalogs: true
I20251026 02:56:38.841782 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7ece4924086a4ae68db59650193f0e0e" } modify_external_catalogs: true
I20251026 02:56:38.843183 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 66ce026fce6e432faaf5f334c8af3f63 (DEFAULT_TABLE table=default.Table_b083b1c076f1492a93b6e8f74e8bc574 [id=de9db409d30f4d07bd5eb62c5c9a046e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.843569 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66ce026fce6e432faaf5f334c8af3f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.843585 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 66ce026fce6e432faaf5f334c8af3f63 (DEFAULT_TABLE table=default.Table_b083b1c076f1492a93b6e8f74e8bc574 [id=de9db409d30f4d07bd5eb62c5c9a046e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.843940 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66ce026fce6e432faaf5f334c8af3f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.848115 2378 tablet_bootstrap.cc:492] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:38.848326 2378 ts_tablet_manager.cc:1403] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.005s
I20251026 02:56:38.849021 2378 raft_consensus.cc:359] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.849221 2378 raft_consensus.cc:385] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.849303 2378 raft_consensus.cc:740] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.849535 2378 consensus_queue.cc:260] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.850900 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_35945e2ea4504c3c8db41203934516df" } modify_external_catalogs: true
I20251026 02:56:38.851004 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f0f7c737d8df4eb09c5408b0b02e954b" } modify_external_catalogs: true
I20251026 02:56:38.851102 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_35945e2ea4504c3c8db41203934516df" } modify_external_catalogs: true
I20251026 02:56:38.851200 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f0f7c737d8df4eb09c5408b0b02e954b" } modify_external_catalogs: true
I20251026 02:56:38.719153 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 1dec51b88e3545d1a0a7793613540ee3 (DEFAULT_TABLE table=default.Table_e6acd68207564173ac53389222b0c621 [id=c6ea9ee3b1a749609a08e993826b74e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.852335 2377 log.cc:1199] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/4011930a277d410f863a1558416c4b0f
I20251026 02:56:38.852586 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dec51b88e3545d1a0a7793613540ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.852730 2378 ts_tablet_manager.cc:1434] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251026 02:56:38.852759 2377 ts_tablet_manager.cc:1950] T 4011930a277d410f863a1558416c4b0f P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:38.853014 2378 tablet_bootstrap.cc:492] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:38.853062 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 527) took 6981 ms. Trace:
I20251026 02:56:38.853142 2377 rpcz_store.cc:276] 1026 02:56:31.872117 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:31.875196 (+ 3079us) service_pool.cc:225] Handling call
1026 02:56:38.635252 (+6760056us) ts_tablet_manager.cc:1151] Deleting tablet 4011930a277d410f863a1558416c4b0f
1026 02:56:38.635260 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.710993 (+ 75733us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.714242 (+ 3249us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:38.853044 (+138802us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6759768}
I20251026 02:56:38.853886 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 4011930a277d410f863a1558416c4b0f (table default.Table_8dcca91177f5452e816e2080bceadb93 [id=30cb7a766d854568892ad3a733d34f49]) successfully deleted
I20251026 02:56:38.725741 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 905541bcc7644c36976cc987dc6a1d3c (DEFAULT_TABLE table=default.Table_95c013f9c71247aeb254654850f8be47 [id=0d6da4b71f4045848df9edd4a684e057]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.856549 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 905541bcc7644c36976cc987dc6a1d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.857718 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fe3e894262cf4a89b3fcb989449d7569" } modify_external_catalogs: true
I20251026 02:56:38.857901 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fe3e894262cf4a89b3fcb989449d7569" } modify_external_catalogs: true
I20251026 02:56:38.729143 2232 tablet_service.cc:1505] Processing CreateTablet for tablet c0e6905e5ac147e3bc3a0073816faf76 (DEFAULT_TABLE table=default.Table_e3e8bac83d124aacb292bdec63fbb5fb [id=4820a040872241fcb74f10b9dcb4dde3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.858486 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e6905e5ac147e3bc3a0073816faf76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.859138 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 19918cb3b7d0450f9de17026f8ea3295 (DEFAULT_TABLE table=default.Table_d00f935fbc4b4d80820018fa2857a22d [id=7708ac42508844ec984ce0f5ac8e95f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.730747 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 4568b52fb082412490aa5fe8da4fdc35 (DEFAULT_TABLE table=default.Table_ed035c4e56264e5c8401998633ac2a04 [id=54273ff4359f42348569f36ff845ca6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.740607 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fe0e06a86e241bc91a1db4219fe25bc" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:38.744822 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8688a0bf946841deb5d88c931538b129" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:38.754027 2234 tablet_service.cc:1505] Processing CreateTablet for tablet a8e8959e8af84742beb00e3ea2f2f4c0 (DEFAULT_TABLE table=default.Table_e74211bb29f145a8aeec9c6f0dab4bff [id=2ffcdca25c58488c849387f50016c42d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.755087 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 18244445760a434ba7ab2f3a974da7a0 (DEFAULT_TABLE table=default.Table_dc22b2aa5fe64641b34fa498992594cc [id=f4513ca6bc3b49d9a8e985d821988ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.782954 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe48472efa444bf580e4074c96a9de92" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:38.860190 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4568b52fb082412490aa5fe8da4fdc35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.859498 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19918cb3b7d0450f9de17026f8ea3295. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.791631 3239 raft_consensus.cc:493] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.797217 2243 tablet_service.cc:1505] Processing CreateTablet for tablet bfd54c88330e4ebba6c07fca53cacc53 (DEFAULT_TABLE table=default.Table_f25a397301e7496a8d70b9a34ae32f6b [id=48e820dac44640fdaa74eb0c1ef4d0e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.861692 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18244445760a434ba7ab2f3a974da7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.861366 2259 raft_consensus.cc:2468] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:38.861896 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfd54c88330e4ebba6c07fca53cacc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.862054 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e8959e8af84742beb00e3ea2f2f4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.861709 3239 raft_consensus.cc:515] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.861012 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 19918cb3b7d0450f9de17026f8ea3295 (DEFAULT_TABLE table=default.Table_d00f935fbc4b4d80820018fa2857a22d [id=7708ac42508844ec984ce0f5ac8e95f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.859628 2378 tablet_bootstrap.cc:654] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.862615 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19918cb3b7d0450f9de17026f8ea3295. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.862802 3239 leader_election.cc:290] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:38.797367 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 02861f399f4d488586f279c01eebcdd1 (DEFAULT_TABLE table=default.Table_f0f7c737d8df4eb09c5408b0b02e954b [id=93c4ab50163c427b8cec6855adb3df48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.863354 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02861f399f4d488586f279c01eebcdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.797631 2226 tablet_service.cc:1505] Processing CreateTablet for tablet df585f11274f4fd2852890aa2c53f0c0 (DEFAULT_TABLE table=default.Table_a463861b69f34f099edf69c1eb7d99a6 [id=58af1b534f644c758e06ca97aa86fce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.864193 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df585f11274f4fd2852890aa2c53f0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.797916 2231 tablet_service.cc:1505] Processing CreateTablet for tablet ade0f7de597049ffb45274d60c9c0932 (DEFAULT_TABLE table=default.Table_b67009abd92345148b7cb2d1be01e4a9 [id=de5ae8e655474ee7b05f3ed272288251]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.864789 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade0f7de597049ffb45274d60c9c0932. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.799312 2229 tablet_service.cc:1505] Processing CreateTablet for tablet a43c7d16330644b5bfa0a88e5b624f6b (DEFAULT_TABLE table=default.Table_20aea86003f649d282631fee5df071c8 [id=8865de0dc99d477f8adc5ae8a3a502ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.868448 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43c7d16330644b5bfa0a88e5b624f6b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:38.869025 2053 leader_election.cc:343] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.869177 2053 leader_election.cc:304] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.869617 2053 leader_election.cc:304] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.870190 3006 raft_consensus.cc:2749] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.870350 3006 raft_consensus.cc:2804] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:38.870399 3006 raft_consensus.cc:493] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:38.870450 3006 raft_consensus.cc:3060] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.806584 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2524ee0c964345459c3e62f1e39b8f9e" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:38.814271 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 97aa7f0266854c0a8f17e6bceac0c9b5 (DEFAULT_TABLE table=default.Table_50ea87134e5c4bba934f9f7a5e275c9b [id=34a9fbe2da4d47e9a600083ae1a85971]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.872488 1915 leader_election.cc:304] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:38.872781 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97aa7f0266854c0a8f17e6bceac0c9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.872874 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 899a951f4f5d429b9a2a41cea2542a45 (DEFAULT_TABLE table=default.Table_e0971fd722424b9cb2a4067114d03cfe [id=c6b7c6aaeda54c0a847f2f85ab78f425]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.873039 3006 raft_consensus.cc:515] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.873211 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899a951f4f5d429b9a2a41cea2542a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.873658 3006 leader_election.cc:290] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.873875 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dba102838945453cbf44e1cfe505a52c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:38.874019 2114 raft_consensus.cc:2468] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:38.818051 2227 tablet_service.cc:1505] Processing CreateTablet for tablet 0aabfe11dedd454f8221fb5de25a268b (DEFAULT_TABLE table=default.Table_6fc619ce52194bd58287fde601a7a24d [id=1b8eca019058481ebe80d9cb89ed1590]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.876564 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aabfe11dedd454f8221fb5de25a268b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.823508 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 0b1f0e8f18c84576a574efd9f46ad458 (DEFAULT_TABLE table=default.Table_09e15e381f1d41dca2fb4079688213db [id=9b6cbcd7ee0c48e0b1d21a00371046b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.880471 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b1f0e8f18c84576a574efd9f46ad458. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.880676 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dba102838945453cbf44e1cfe505a52c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:38.881698 2195 leader_election.cc:304] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:38.883118 2094 tablet_service.cc:1505] Processing CreateTablet for tablet a791991448b7484588cf6e501f68b3cf (DEFAULT_TABLE table=default.Table_34405bccc46d4867acc88741694d9255 [id=410c8f96bf2b47e597bb8dfd76c18d74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.883299 3517 raft_consensus.cc:2804] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:38.883381 3517 raft_consensus.cc:493] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:38.883451 3517 raft_consensus.cc:3060] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.883507 1915 leader_election.cc:304] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:38.883556 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8688a0bf946841deb5d88c931538b129" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:38.883744 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a791991448b7484588cf6e501f68b3cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:38.883951 2194 leader_election.cc:343] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:38.884882 2053 leader_election.cc:343] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.885947 3517 raft_consensus.cc:515] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.886518 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 2e4c8c2d3d0945e4be8ee65f152b7bd9 (DEFAULT_TABLE table=default.Table_c861702df3424cdba2d706ea025a6529 [id=245494c49f044285bfe54e828e0e1553]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.886752 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 899a951f4f5d429b9a2a41cea2542a45 (DEFAULT_TABLE table=default.Table_e0971fd722424b9cb2a4067114d03cfe [id=c6b7c6aaeda54c0a847f2f85ab78f425]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.886888 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4c8c2d3d0945e4be8ee65f152b7bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.887091 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899a951f4f5d429b9a2a41cea2542a45. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:38.887818 2195 consensus_peers.cc:597] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efd751ee4b1d433198dcd00c081f648d. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:38.888363 3372 raft_consensus.cc:2749] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.888516 3372 raft_consensus.cc:2749] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:38.833405 2238 tablet_service.cc:1505] Processing CreateTablet for tablet 992a43f2dabf4e13865e4c128aae763e (DEFAULT_TABLE table=default.Table_6b7d3c3d1bbc40f9b1ef1a34f2f5332f [id=75a2e41b0d134dcdb1c0bc3ea474c467]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.890219 2242 tablet_service.cc:1505] Processing CreateTablet for tablet a791991448b7484588cf6e501f68b3cf (DEFAULT_TABLE table=default.Table_34405bccc46d4867acc88741694d9255 [id=410c8f96bf2b47e597bb8dfd76c18d74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.724581 2381 tablet_bootstrap.cc:654] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.890591 3517 leader_election.cc:290] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:38.890792 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8688a0bf946841deb5d88c931538b129" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:38.890916 2256 raft_consensus.cc:3060] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.891139 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dba102838945453cbf44e1cfe505a52c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:38.891307 2114 raft_consensus.cc:3060] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.893530 2256 raft_consensus.cc:2468] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:38.890317 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 992a43f2dabf4e13865e4c128aae763e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.894193 2114 raft_consensus.cc:2468] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:38.894817 3510 consensus_queue.cc:1048] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:38.896035 2195 leader_election.cc:304] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:38.896482 2053 leader_election.cc:304] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:38.896957 3510 raft_consensus.cc:2804] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:38.897082 3510 raft_consensus.cc:697] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:38.897372 3510 consensus_queue.cc:237] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.899446 3517 raft_consensus.cc:2804] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:38.843461 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 66ce026fce6e432faaf5f334c8af3f63 (DEFAULT_TABLE table=default.Table_b083b1c076f1492a93b6e8f74e8bc574 [id=de9db409d30f4d07bd5eb62c5c9a046e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.899608 3517 raft_consensus.cc:697] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:38.899919 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66ce026fce6e432faaf5f334c8af3f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.899825 3517 consensus_queue.cc:237] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.902797 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a791991448b7484588cf6e501f68b3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.905529 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dba102838945453cbf44e1cfe505a52c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:38.907522 2194 leader_election.cc:343] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:38.916882 2381 tablet_bootstrap.cc:492] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:38.917025 2381 ts_tablet_manager.cc:1403] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.279s user 0.009s sys 0.001s
I20251026 02:56:38.917682 2381 raft_consensus.cc:359] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.917838 2381 raft_consensus.cc:385] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.917878 2381 raft_consensus.cc:740] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.918079 2381 consensus_queue.cc:260] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.919296 2381 ts_tablet_manager.cc:1434] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:38.922088 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 9b9dc0825e0844f68d5afcc201db737c (DEFAULT_TABLE table=default.Table_5a7f5adeea764cda87c60443f2e29e50 [id=091e1ee9ed8345d897123bb4abe7391c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.922529 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9dc0825e0844f68d5afcc201db737c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.922734 2089 tablet_service.cc:1505] Processing CreateTablet for tablet 9b9dc0825e0844f68d5afcc201db737c (DEFAULT_TABLE table=default.Table_5a7f5adeea764cda87c60443f2e29e50 [id=091e1ee9ed8345d897123bb4abe7391c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.923166 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9dc0825e0844f68d5afcc201db737c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.924050 2099 tablet_service.cc:1505] Processing CreateTablet for tablet f2bae845251944d780b06be90cd659e5 (DEFAULT_TABLE table=default.Table_901960ec4b4749f7b73a558f65408857 [id=232e25932feb4f6bbb665815b97899cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.924438 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2bae845251944d780b06be90cd659e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.924816 1948 tablet_service.cc:1505] Processing CreateTablet for tablet f2bae845251944d780b06be90cd659e5 (DEFAULT_TABLE table=default.Table_901960ec4b4749f7b73a558f65408857 [id=232e25932feb4f6bbb665815b97899cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.924930 2381 tablet_bootstrap.cc:492] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:38.925149 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2bae845251944d780b06be90cd659e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.926337 2381 tablet_bootstrap.cc:654] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:38.922734 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 2e4c8c2d3d0945e4be8ee65f152b7bd9 (DEFAULT_TABLE table=default.Table_c861702df3424cdba2d706ea025a6529 [id=245494c49f044285bfe54e828e0e1553]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.931186 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4c8c2d3d0945e4be8ee65f152b7bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.932017 2229 tablet_service.cc:1505] Processing CreateTablet for tablet f2bae845251944d780b06be90cd659e5 (DEFAULT_TABLE table=default.Table_901960ec4b4749f7b73a558f65408857 [id=232e25932feb4f6bbb665815b97899cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.932433 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2bae845251944d780b06be90cd659e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.932711 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 2e4c8c2d3d0945e4be8ee65f152b7bd9 (DEFAULT_TABLE table=default.Table_c861702df3424cdba2d706ea025a6529 [id=245494c49f044285bfe54e828e0e1553]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.932996 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4c8c2d3d0945e4be8ee65f152b7bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.934160 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 9b9dc0825e0844f68d5afcc201db737c (DEFAULT_TABLE table=default.Table_5a7f5adeea764cda87c60443f2e29e50 [id=091e1ee9ed8345d897123bb4abe7391c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.935026 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 519d1ae6300b488ca0bd99d380bffcea (DEFAULT_TABLE table=default.Table_a4383df3509649cb824fea52dd98449e [id=aadd07d90ec24e60b715942513febff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.935397 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519d1ae6300b488ca0bd99d380bffcea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.936897 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 899a951f4f5d429b9a2a41cea2542a45 (DEFAULT_TABLE table=default.Table_e0971fd722424b9cb2a4067114d03cfe [id=c6b7c6aaeda54c0a847f2f85ab78f425]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.937254 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899a951f4f5d429b9a2a41cea2542a45. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:38.939436 2053 consensus_peers.cc:597] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:38.940685 1947 tablet_service.cc:1505] Processing CreateTablet for tablet a791991448b7484588cf6e501f68b3cf (DEFAULT_TABLE table=default.Table_34405bccc46d4867acc88741694d9255 [id=410c8f96bf2b47e597bb8dfd76c18d74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.941074 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a791991448b7484588cf6e501f68b3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.942960 2086 tablet_service.cc:1505] Processing CreateTablet for tablet cfa2b17eb62646d5920e43d9200477a5 (DEFAULT_TABLE table=default.Table_906d1b8fb76f4fd69325086bafba430d [id=fde430825b5e46e58402e1913f583966]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.943427 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfa2b17eb62646d5920e43d9200477a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.944862 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9dc0825e0844f68d5afcc201db737c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.944923 1978 raft_consensus.cc:3060] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.952579 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 46547883cfe041f3954aa4128fba1d8c
I20251026 02:56:38.952833 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 02861f399f4d488586f279c01eebcdd1
I20251026 02:56:38.953996 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 46547883cfe041f3954aa4128fba1d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.954591 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3e424a9c89e64b1c98ada17fa5c31e9e
I20251026 02:56:38.955619 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e424a9c89e64b1c98ada17fa5c31e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.956007 1660 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet d99dedf78d8247d4be8760028d8f3fa0
I20251026 02:56:38.956830 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 02861f399f4d488586f279c01eebcdd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:38.958695 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 519d1ae6300b488ca0bd99d380bffcea (DEFAULT_TABLE table=default.Table_a4383df3509649cb824fea52dd98449e [id=aadd07d90ec24e60b715942513febff9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:38.958987 2194 consensus_peers.cc:597] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:38.799253 1954 tablet_service.cc:1505] Processing CreateTablet for tablet ade0f7de597049ffb45274d60c9c0932 (DEFAULT_TABLE table=default.Table_b67009abd92345148b7cb2d1be01e4a9 [id=de5ae8e655474ee7b05f3ed272288251]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.962891 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade0f7de597049ffb45274d60c9c0932. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.963310 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 519d1ae6300b488ca0bd99d380bffcea (DEFAULT_TABLE table=default.Table_a4383df3509649cb824fea52dd98449e [id=aadd07d90ec24e60b715942513febff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.798915 2376 raft_consensus.cc:2243] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:38.967567 2376 raft_consensus.cc:2272] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:38.967593 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 92b238ee894e4a16b3609784910266b4 (DEFAULT_TABLE table=default.Table_1f67f077a0ce4edab1c2711aa4d9556c [id=11cbf7018d894c20a980838e54ccae53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.968022 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92b238ee894e4a16b3609784910266b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.799584 1951 tablet_service.cc:1505] Processing CreateTablet for tablet a43c7d16330644b5bfa0a88e5b624f6b (DEFAULT_TABLE table=default.Table_20aea86003f649d282631fee5df071c8 [id=8865de0dc99d477f8adc5ae8a3a502ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.968416 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43c7d16330644b5bfa0a88e5b624f6b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:38.974627 1629 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 201992us
I20251026 02:56:38.976789 2095 tablet_service.cc:1505] Processing CreateTablet for tablet dd7f62a7ea5344fcb1ac8499278aaec4 (DEFAULT_TABLE table=default.Table_138fc883cb304daeb0fc59f09921d7c2 [id=17090cdf8c5346b9bb0630b1beeb5e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.977421 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7f62a7ea5344fcb1ac8499278aaec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.986079 3372 raft_consensus.cc:493] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:38.986249 3372 raft_consensus.cc:515] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:38.986932 3372 leader_election.cc:290] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:38.988471 2381 tablet_bootstrap.cc:492] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:38.988617 2381 ts_tablet_manager.cc:1403] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.002s
W20251026 02:56:38.989378 1916 leader_election.cc:343] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 0de06ea2b0544402a979d9ddc487860f
I20251026 02:56:38.989435 2381 raft_consensus.cc:359] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.989639 2381 raft_consensus.cc:385] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:38.989688 2381 raft_consensus.cc:740] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:38.989861 2381 consensus_queue.cc:260] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:38.989910 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 53264ef08ff7454faa19b18fa35bd803 (DEFAULT_TABLE table=default.Table_79f94fa4977d4066907d585cd301d523 [id=50fc2dd7098d493c816c4d379d73647f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:38.990324 2194 consensus_peers.cc:597] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:38.990923 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53264ef08ff7454faa19b18fa35bd803. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:38.996987 1974 raft_consensus.cc:3060] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:38.962229 1956 tablet_service.cc:1505] Processing CreateTablet for tablet cfa2b17eb62646d5920e43d9200477a5 (DEFAULT_TABLE table=default.Table_906d1b8fb76f4fd69325086bafba430d [id=fde430825b5e46e58402e1913f583966]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.998845 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfa2b17eb62646d5920e43d9200477a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.000494 2378 tablet_bootstrap.cc:492] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:39.000654 2378 ts_tablet_manager.cc:1403] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.148s user 0.002s sys 0.009s
I20251026 02:56:39.001092 2376 ts_tablet_manager.cc:1916] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.001479 2378 raft_consensus.cc:359] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.003269 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 (DEFAULT_TABLE table=default.Table_10236dd7e7264f888b32afcb8c56c95f [id=5ca881edae84459c8889cbf141b18d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.002554 1978 raft_consensus.cc:3060] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.003552 2378 raft_consensus.cc:385] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.003815 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.003686 2378 raft_consensus.cc:740] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.004244 2378 consensus_queue.cc:260] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.004781 2378 ts_tablet_manager.cc:1434] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251026 02:56:39.004730 3537 consensus_queue.cc:1048] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:39.005026 2378 tablet_bootstrap.cc:492] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:39.005417 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 97aa7f0266854c0a8f17e6bceac0c9b5 (DEFAULT_TABLE table=default.Table_50ea87134e5c4bba934f9f7a5e275c9b [id=34a9fbe2da4d47e9a600083ae1a85971]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:38.990329 2381 ts_tablet_manager.cc:1434] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:39.006966 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 46547883cfe041f3954aa4128fba1d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.007206 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 02861f399f4d488586f279c01eebcdd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.007519 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519d1ae6300b488ca0bd99d380bffcea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.008801 2231 tablet_service.cc:1505] Processing CreateTablet for tablet cfa2b17eb62646d5920e43d9200477a5 (DEFAULT_TABLE table=default.Table_906d1b8fb76f4fd69325086bafba430d [id=fde430825b5e46e58402e1913f583966]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.009202 2227 tablet_service.cc:1552] Processing DeleteTablet for tablet 02861f399f4d488586f279c01eebcdd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.009431 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 46547883cfe041f3954aa4128fba1d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.011159 2378 tablet_bootstrap.cc:654] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.013051 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 7a45d350980046df88c5896e1f3dffff (DEFAULT_TABLE table=default.Table_4ded23ba48e4400b9b96e1dd82638b49 [id=a0a7c40e6bc240e6b741a83c3e7ccf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.013464 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a45d350980046df88c5896e1f3dffff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.015974 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519d1ae6300b488ca0bd99d380bffcea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.016928 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 533) took 6824 ms. Trace:
I20251026 02:56:39.017058 2377 rpcz_store.cc:276] 1026 02:56:32.192812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.193257 (+ 445us) service_pool.cc:225] Handling call
1026 02:56:38.989278 (+6796021us) ts_tablet_manager.cc:1151] Deleting tablet 9fb42b54f558499090a1d7cf257dd84c
1026 02:56:39.016759 (+ 27481us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.016910 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6795831}
I20251026 02:56:39.017522 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 534) took 6824 ms. Trace:
I20251026 02:56:39.017624 2377 rpcz_store.cc:276] 1026 02:56:32.192903 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.193483 (+ 580us) service_pool.cc:225] Handling call
1026 02:56:39.017382 (+6823899us) ts_tablet_manager.cc:1151] Deleting tablet 21b936fa4f6a4ff6af321971a30cccbe
1026 02:56:39.017393 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.017506 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6823732}
I20251026 02:56:39.017978 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 535) took 6825 ms. Trace:
I20251026 02:56:39.018075 2377 rpcz_store.cc:276] 1026 02:56:32.192994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.193688 (+ 694us) service_pool.cc:225] Handling call
1026 02:56:39.017860 (+6824172us) ts_tablet_manager.cc:1151] Deleting tablet 027dd92fc5cf4c9e847de3e55e0dac1c
1026 02:56:39.017867 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.017961 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6823965}
I20251026 02:56:39.018182 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97aa7f0266854c0a8f17e6bceac0c9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.018421 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 536) took 6825 ms. Trace:
I20251026 02:56:39.018512 2377 rpcz_store.cc:276] 1026 02:56:32.193107 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.193922 (+ 815us) service_pool.cc:225] Handling call
1026 02:56:39.018318 (+6824396us) ts_tablet_manager.cc:1151] Deleting tablet 4a3f3fc3c7a04d05bae28472251f3acc
1026 02:56:39.018326 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.018406 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6824249}
I20251026 02:56:39.018803 2377 tablet_replica.cc:333] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:39.019017 2377 raft_consensus.cc:2243] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:39.019382 2377 raft_consensus.cc:2272] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.019929 2377 ts_tablet_manager.cc:1916] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.021200 2381 tablet_bootstrap.cc:492] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:39.021235 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfa2b17eb62646d5920e43d9200477a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.029135 2372 ts_tablet_manager.cc:1929] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:39.036741 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e424a9c89e64b1c98ada17fa5c31e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.037897 2381 tablet_bootstrap.cc:654] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.045346 3372 raft_consensus.cc:493] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.045493 3372 raft_consensus.cc:515] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.046075 3372 leader_election.cc:290] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:39.047480 3372 raft_consensus.cc:493] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.047616 3372 raft_consensus.cc:515] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.048267 3372 leader_election.cc:290] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:39.049209 1916 leader_election.cc:343] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: efd751ee4b1d433198dcd00c081f648d
W20251026 02:56:39.054276 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 9fb42b54f558499090a1d7cf257dd84c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fb42b54f558499090a1d7cf257dd84c already in progress: creating tablet
I20251026 02:56:39.062433 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c61f3600fe8e4e81af872d53e8973f25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:39.071487 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c61f3600fe8e4e81af872d53e8973f25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.062871 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_51cf6f44f72b4fea9695ac3880595e80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:39.071883 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51cf6f44f72b4fea9695ac3880595e80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.063292 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_731fe2201e1f483c9997355df8e8b672"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:39.063664 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b68d07c85de14db79377412aa4474098"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:39.063992 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_56366720e640479392e1652794ea3847"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:39.064499 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_38943948af334d7b907bfbd6cd428fd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:39.064754 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a2e4dfe2f01f4d8fac1b89211f87d9a6" } new_table_name: "default.Table_37a214cdf9c846cf8e0f55e6cbd3d274" modify_external_catalogs: true
W20251026 02:56:39.072625 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_731fe2201e1f483c9997355df8e8b672 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:39.072935 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38943948af334d7b907bfbd6cd428fd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:39.073177 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b68d07c85de14db79377412aa4474098 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:39.073446 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56366720e640479392e1652794ea3847 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:39.078308 2194 consensus_peers.cc:597] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:39.081884 2194 consensus_peers.cc:597] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a821ed8a686c452a975acb96b5c74778. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:39.029469 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 0c60d8227eed417595ef68d44c68ebcf (DEFAULT_TABLE table=default.Table_09b68d3ebab94c55b3c90567af20db3d [id=d35f10fbbafe4c1595eaf1a17c4c091e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.086227 2372 log.cc:1199] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3327d26a2c2c4d0b86d98151f5940bd7
I20251026 02:56:39.034036 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 6294c8954f9749f998d5ae1c856e02ff (DEFAULT_TABLE table=default.Table_fa22c9dc29814d9cb5c4bfdec072a3cf [id=a111da525efe4df2a129972120a3871a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.031047 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 3131bc2948e34c25a1bbda90f8dabeb3 (DEFAULT_TABLE table=default.Table_ce57d8a514194198ad913d3eb35269b9 [id=65f99e4eec4044b9b9a12835401e111e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.035962 2094 tablet_service.cc:1505] Processing CreateTablet for tablet a32b1a6b856c47b993a308db94dac357 (DEFAULT_TABLE table=default.Table_111996853fae40db821e68942c9fae5e [id=e58df69dfe05472a96acaedee1928294]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.043416 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 2e0c6a79658640c59ca9d026e3c24933 (DEFAULT_TABLE table=default.Table_63579add544846d3b2a9f267e24fda8f [id=31ab101040634eefb3abc80b3b052054]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.045858 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 11f4c327caa644d89098496bc324b976 (DEFAULT_TABLE table=default.Table_914472564539427caff89590b14d9d01 [id=fff9b966161444b990693ad4adbfcbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.046629 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7da859119bd42ea9fa1290b3d4bf88c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:39.051926 2091 tablet_service.cc:1505] Processing CreateTablet for tablet c58ce24eb8f24eee900b81bacc2a1ac3 (DEFAULT_TABLE table=default.Table_bb11d644fbd64beb9783fd730f8bb577 [id=21aff80020ca401fad4685acc5a23266]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.060294 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 28f529b6c18d412e8caa10753eab4e01 (DEFAULT_TABLE table=default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.063426 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 5eda85f2d53a43d68907c467c7da9bd6 (DEFAULT_TABLE table=default.Table_517fcc0e025a4b8f9ec1a392735037a9 [id=89a2069d105b44339fe06023cf322fa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.036367 3587 raft_consensus.cc:493] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.087194 3587 raft_consensus.cc:515] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.088124 3587 leader_election.cc:290] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:39.088387 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:39.088567 2194 consensus_peers.cc:597] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:39.089012 2053 leader_election.cc:343] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.089249 2372 ts_tablet_manager.cc:1950] T 3327d26a2c2c4d0b86d98151f5940bd7 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:39.089468 2194 consensus_peers.cc:597] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:39.090554 2375 tablet_bootstrap.cc:492] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:39.090700 2375 ts_tablet_manager.cc:1403] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.303s user 0.003s sys 0.004s
I20251026 02:56:39.091163 2381 tablet_bootstrap.cc:492] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:39.091598 2375 raft_consensus.cc:359] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.091753 2375 raft_consensus.cc:385] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.091826 2375 raft_consensus.cc:740] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.092129 2375 consensus_queue.cc:260] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.092691 2381 ts_tablet_manager.cc:1403] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.001s
I20251026 02:56:39.092936 2375 ts_tablet_manager.cc:1434] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:39.084650 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 7bef60e3a10843b4be2022ab4b64be68 (DEFAULT_TABLE table=default.Table_3788e42710a0457c80f6ae601866b2ee [id=edb0d561b44848b0976b3a7d0d52b41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.091905 1982 raft_consensus.cc:3060] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.074481 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 19918cb3b7d0450f9de17026f8ea3295 (DEFAULT_TABLE table=default.Table_d00f935fbc4b4d80820018fa2857a22d [id=7708ac42508844ec984ce0f5ac8e95f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.098810 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b819b43894574dc9ad52b4081e46628e
I20251026 02:56:39.099644 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0b10d09d420e46ac9a13bada7fd2f740"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:39.100235 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b10d09d420e46ac9a13bada7fd2f740 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.102293 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet b819b43894574dc9ad52b4081e46628e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:39.103847 2381 raft_consensus.cc:359] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.109246 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet 7053d454def04264a956aa75ceaad5c9 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:39.110456 2381 raft_consensus.cc:385] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.112123 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_55e6f4711fcc48eda34b1f7b62a7ad2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:39.112519 2381 raft_consensus.cc:740] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
W20251026 02:56:39.112728 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55e6f4711fcc48eda34b1f7b62a7ad2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.112860 2381 consensus_queue.cc:260] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.113481 2381 ts_tablet_manager.cc:1434] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20251026 02:56:39.113852 2381 tablet_bootstrap.cc:492] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:39.114292 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fef50bc3bf28484f9b1c67ab84027952"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:39.114966 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fef50bc3bf28484f9b1c67ab84027952 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:39.115533 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 868dbbfce40c4dd6b36cc220966b1986 consensus state has no leader
I20251026 02:56:39.115970 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c96a0fe4ea684dd29d1efeaab923e539" } new_table_name: "default.Table_e6b19dda39ad45baae642c27b07db42d" modify_external_catalogs: true
I20251026 02:56:39.121698 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c60d8227eed417595ef68d44c68ebcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.127547 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7053d454def04264a956aa75ceaad5c9
I20251026 02:56:39.128364 1659 catalog_manager.cc:6719] ReplaceTablet: tablet 7053d454def04264a956aa75ceaad5c9 deleted and replaced by tablet 63eac91f5cb540e6b999ac816ee0b89d
I20251026 02:56:39.128396 2375 tablet_bootstrap.cc:492] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:39.129865 2375 tablet_bootstrap.cc:654] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:39.130043 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4b0933f71cb4ac7a58986d16b613e64 consensus state has no leader
I20251026 02:56:39.128846 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0ce6582d31c14fdfb11fa3342b1981bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:39.130450 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_74bad7faa1614c0fa737ab4293efa31c" } new_table_name: "default.Table_c84ad81d49374f2aab22ef0c241af795" modify_external_catalogs: true
W20251026 02:56:39.131037 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ce6582d31c14fdfb11fa3342b1981bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.131188 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6294c8954f9749f998d5ae1c856e02ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.132148 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_63579add544846d3b2a9f267e24fda8f" } new_table_name: "default.Table_39b07405b7ef45e58339e61108e7b9e9" modify_external_catalogs: true
I20251026 02:56:39.133459 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_42e3536941484e83a992e13086dc316e" } new_table_name: "default.Table_3af0e3e691254fbb8f124697619830bf" modify_external_catalogs: true
I20251026 02:56:39.136116 1662 catalog_manager.cc:5649] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:39.136458 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a32b1a6b856c47b993a308db94dac357. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.136523 1662 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet dba102838945453cbf44e1cfe505a52c (table default.Table_8ffce6cf6b2d44aa8a9459e834de1d80 [id=92310ee99b5c428c9773b0f7a5462a4d]). Expected version 1 got 0
I20251026 02:56:39.140460 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e0c6a79658640c59ca9d026e3c24933. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.144675 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f4c327caa644d89098496bc324b976. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.148314 2381 tablet_bootstrap.cc:654] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.148378 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c58ce24eb8f24eee900b81bacc2a1ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.152426 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eda85f2d53a43d68907c467c7da9bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.156440 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3131bc2948e34c25a1bbda90f8dabeb3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:39.156929 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f197526e7d34f849dbb4f28d8d414cf consensus state has no leader
I20251026 02:56:39.160488 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f529b6c18d412e8caa10753eab4e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.174714 2376 ts_tablet_manager.cc:1929] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:39.174851 2376 log.cc:1199] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/ca4611c0a494425ca50d599b310872c8
I20251026 02:56:39.175151 2376 ts_tablet_manager.cc:1950] T ca4611c0a494425ca50d599b310872c8 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:39.175496 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 528) took 5754 ms. Trace:
I20251026 02:56:39.175577 2376 rpcz_store.cc:276] 1026 02:56:33.421252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.421567 (+ 315us) service_pool.cc:225] Handling call
1026 02:56:38.781918 (+5360351us) ts_tablet_manager.cc:1151] Deleting tablet ca4611c0a494425ca50d599b310872c8
1026 02:56:38.781928 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.076404 (+294476us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.174627 (+ 98223us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.175476 (+ 849us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5144,"tablet-delete.queue_time_us":5360064}
I20251026 02:56:39.175875 2381 tablet_bootstrap.cc:492] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:39.175982 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 529) took 5714 ms. Trace:
I20251026 02:56:39.176003 2381 ts_tablet_manager.cc:1403] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.062s user 0.000s sys 0.002s
I20251026 02:56:39.176047 2376 rpcz_store.cc:276] 1026 02:56:33.461309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.461381 (+ 72us) service_pool.cc:225] Handling call
1026 02:56:39.175859 (+5714478us) ts_tablet_manager.cc:1151] Deleting tablet 1fb2ff87cf0248a0a4a77f2733ef867b
1026 02:56:39.175873 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.175968 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5714113}
I20251026 02:56:39.176359 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 531) took 5685 ms. Trace:
I20251026 02:56:39.176425 2376 rpcz_store.cc:276] 1026 02:56:33.491153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.491433 (+ 280us) service_pool.cc:225] Handling call
1026 02:56:39.176263 (+5684830us) ts_tablet_manager.cc:1151] Deleting tablet 3e38f282e8414b7db2a6d5f4a8933e84
1026 02:56:39.176274 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.176346 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5684559}
I20251026 02:56:39.176679 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 532) took 5685 ms. Trace:
I20251026 02:56:39.176734 2376 rpcz_store.cc:276] 1026 02:56:33.491259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.491748 (+ 489us) service_pool.cc:225] Handling call
1026 02:56:39.176598 (+5684850us) ts_tablet_manager.cc:1151] Deleting tablet 892a5fab2af1432eb31d92d9859ea2a0
1026 02:56:39.176605 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.176668 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5684651}
I20251026 02:56:39.176724 2381 raft_consensus.cc:359] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.176891 2381 raft_consensus.cc:385] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.176946 2381 raft_consensus.cc:740] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.176970 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 533) took 5685 ms. Trace:
I20251026 02:56:39.177027 2376 rpcz_store.cc:276] 1026 02:56:33.491365 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.491422 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:39.176891 (+5685469us) ts_tablet_manager.cc:1151] Deleting tablet d0f1d6967931482e9de40b0844da6315
1026 02:56:39.176898 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.176959 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5684841}
I20251026 02:56:39.177137 2381 consensus_queue.cc:260] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.177309 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 530) took 5686 ms. Trace:
I20251026 02:56:39.177373 2376 rpcz_store.cc:276] 1026 02:56:33.490990 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.492518 (+ 1528us) service_pool.cc:225] Handling call
1026 02:56:39.177227 (+5684709us) ts_tablet_manager.cc:1151] Deleting tablet 34c6a7c774044411a4733b115080c4c6
1026 02:56:39.177235 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.177297 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5684478}
I20251026 02:56:39.177706 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 540) took 5582 ms. Trace:
I20251026 02:56:39.177742 2381 ts_tablet_manager.cc:1434] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:39.177764 2376 rpcz_store.cc:276] 1026 02:56:33.595324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.595371 (+ 47us) service_pool.cc:225] Handling call
1026 02:56:39.177624 (+5582253us) ts_tablet_manager.cc:1151] Deleting tablet ed836bb7519e4b0290e6547fa57128bc
1026 02:56:39.177633 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.177694 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5582055}
I20251026 02:56:39.177985 2376 tablet_replica.cc:333] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:39.178146 2376 raft_consensus.cc:2243] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:39.178305 2376 raft_consensus.cc:2272] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.178444 2381 tablet_bootstrap.cc:492] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:39.180325 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 126067us
W20251026 02:56:39.180621 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 21b936fa4f6a4ff6af321971a30cccbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21b936fa4f6a4ff6af321971a30cccbe already in progress: creating tablet
W20251026 02:56:39.180882 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 027dd92fc5cf4c9e847de3e55e0dac1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 027dd92fc5cf4c9e847de3e55e0dac1c already in progress: creating tablet
W20251026 02:56:39.181053 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 4a3f3fc3c7a04d05bae28472251f3acc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a3f3fc3c7a04d05bae28472251f3acc already in progress: creating tablet
I20251026 02:56:39.182044 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet ca4611c0a494425ca50d599b310872c8 (table default.Table_21d71fb70373400396a75b6e52416078 [id=2dcd0d98a1354635a3673182e6fad58a]) successfully deleted
W20251026 02:56:39.182301 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 1fb2ff87cf0248a0a4a77f2733ef867b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fb2ff87cf0248a0a4a77f2733ef867b already in progress: creating tablet
W20251026 02:56:39.182536 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3e38f282e8414b7db2a6d5f4a8933e84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e38f282e8414b7db2a6d5f4a8933e84 already in progress: creating tablet
W20251026 02:56:39.182772 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 892a5fab2af1432eb31d92d9859ea2a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 892a5fab2af1432eb31d92d9859ea2a0 already in progress: creating tablet
W20251026 02:56:39.182982 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet d0f1d6967931482e9de40b0844da6315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0f1d6967931482e9de40b0844da6315 already in progress: creating tablet
I20251026 02:56:39.183032 2381 tablet_bootstrap.cc:654] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:39.183264 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 34c6a7c774044411a4733b115080c4c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34c6a7c774044411a4733b115080c4c6 already in progress: creating tablet
W20251026 02:56:39.183452 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ed836bb7519e4b0290e6547fa57128bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed836bb7519e4b0290e6547fa57128bc already in progress: creating tablet
I20251026 02:56:39.183625 2376 ts_tablet_manager.cc:1916] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.185035 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.185458 2086 tablet_service.cc:1552] Processing DeleteTablet for tablet 7053d454def04264a956aa75ceaad5c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 63eac91f5cb540e6b999ac816ee0b89d) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:39.189409 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e0c6a79658640c59ca9d026e3c24933 consensus state has no leader
W20251026 02:56:39.189541 2194 consensus_peers.cc:597] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:39.113902 2377 ts_tablet_manager.cc:1929] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:39.193441 2377 log.cc:1199] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/63a63b796191465aab9a63d3ffb0554d
I20251026 02:56:39.194296 2377 ts_tablet_manager.cc:1950] T 63a63b796191465aab9a63d3ffb0554d P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:39.205276 2381 tablet_bootstrap.cc:492] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:39.205565 2381 ts_tablet_manager.cc:1403] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.001s
I20251026 02:56:39.205895 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 537) took 7012 ms. Trace:
I20251026 02:56:39.206383 2381 raft_consensus.cc:359] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.207147 2381 raft_consensus.cc:385] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.207252 2381 raft_consensus.cc:740] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.207543 2381 consensus_queue.cc:260] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.208256 2381 ts_tablet_manager.cc:1434] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 02:56:39.208595 2381 tablet_bootstrap.cc:492] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:39.208563 2053 leader_election.cc:343] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.208722 2053 leader_election.cc:304] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.209163 3587 raft_consensus.cc:2749] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.212391 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet 21b936fa4f6a4ff6af321971a30cccbe with delete_type TABLET_DATA_DELETED (replaced by tablet a956ebfb1eaa4d6fb1575b4165293193) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:39.212632 2381 tablet_bootstrap.cc:654] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.213282 1978 raft_consensus.cc:3060] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.214138 2376 ts_tablet_manager.cc:1929] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:39.214257 2376 log.cc:1199] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/049a915b35224cdaa38df77c734a0f01
I20251026 02:56:39.214630 2376 ts_tablet_manager.cc:1950] T 049a915b35224cdaa38df77c734a0f01 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:39.214728 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bef60e3a10843b4be2022ab4b64be68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.214943 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 545) took 5583 ms. Trace:
I20251026 02:56:39.215013 2376 rpcz_store.cc:276] 1026 02:56:33.631365 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.632072 (+ 707us) service_pool.cc:225] Handling call
1026 02:56:39.177919 (+5545847us) ts_tablet_manager.cc:1151] Deleting tablet 049a915b35224cdaa38df77c734a0f01
1026 02:56:39.177926 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.199333 (+ 21407us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.214010 (+ 14677us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.214926 (+ 916us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5545599}
I20251026 02:56:39.219297 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 565) took 5768 ms. Trace:
I20251026 02:56:39.219483 2372 rpcz_store.cc:276] 1026 02:56:33.450686 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.451499 (+ 813us) service_pool.cc:225] Handling call
1026 02:56:38.703631 (+5252132us) ts_tablet_manager.cc:1151] Deleting tablet 3327d26a2c2c4d0b86d98151f5940bd7
1026 02:56:38.703814 (+ 183us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:38.856018 (+152204us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.028990 (+172972us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.219273 (+190283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5251816}
I20251026 02:56:39.224813 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3327d26a2c2c4d0b86d98151f5940bd7 (table default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 [id=7de20c0869584b56b936144d62202444]) successfully deleted
I20251026 02:56:39.224958 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19918cb3b7d0450f9de17026f8ea3295. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.225719 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 566) took 5774 ms. Trace:
I20251026 02:56:39.225857 2372 rpcz_store.cc:276] 1026 02:56:33.450813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.451655 (+ 842us) service_pool.cc:225] Handling call
1026 02:56:39.225615 (+5773960us) ts_tablet_manager.cc:1151] Deleting tablet 03da415f3a944a9a9679b3a45295ce35
1026 02:56:39.225624 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.225697 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5773809}
W20251026 02:56:39.226347 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 03da415f3a944a9a9679b3a45295ce35 because the tablet was not found. No further retry: Not found: Tablet not found: 03da415f3a944a9a9679b3a45295ce35
I20251026 02:56:39.226763 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 567) took 5757 ms. Trace:
I20251026 02:56:39.226917 2372 rpcz_store.cc:276] 1026 02:56:33.469600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.469677 (+ 77us) service_pool.cc:225] Handling call
1026 02:56:39.226649 (+5756972us) ts_tablet_manager.cc:1151] Deleting tablet 1fb2ff87cf0248a0a4a77f2733ef867b
1026 02:56:39.226662 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.226745 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5756781}
W20251026 02:56:39.227109 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 343606us
W20251026 02:56:39.227406 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 1fb2ff87cf0248a0a4a77f2733ef867b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fb2ff87cf0248a0a4a77f2733ef867b already in progress: creating tablet
I20251026 02:56:39.227921 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 569) took 5741 ms. Trace:
I20251026 02:56:39.228025 2372 rpcz_store.cc:276] 1026 02:56:33.486845 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.487263 (+ 418us) service_pool.cc:225] Handling call
1026 02:56:39.227830 (+5740567us) ts_tablet_manager.cc:1151] Deleting tablet 3e38f282e8414b7db2a6d5f4a8933e84
1026 02:56:39.227839 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.227907 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5740284}
I20251026 02:56:39.228457 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 570) took 5741 ms. Trace:
W20251026 02:56:39.228525 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3e38f282e8414b7db2a6d5f4a8933e84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e38f282e8414b7db2a6d5f4a8933e84 already in progress: creating tablet
I20251026 02:56:39.228580 2372 rpcz_store.cc:276] 1026 02:56:33.486993 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.487632 (+ 639us) service_pool.cc:225] Handling call
1026 02:56:39.228353 (+5740721us) ts_tablet_manager.cc:1151] Deleting tablet 892a5fab2af1432eb31d92d9859ea2a0
1026 02:56:39.228370 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.228442 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5740551}
I20251026 02:56:39.228948 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 571) took 5741 ms. Trace:
I20251026 02:56:39.229063 2372 rpcz_store.cc:276] 1026 02:56:33.487137 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.487897 (+ 760us) service_pool.cc:225] Handling call
1026 02:56:39.228860 (+5740963us) ts_tablet_manager.cc:1151] Deleting tablet d0f1d6967931482e9de40b0844da6315
1026 02:56:39.228868 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.228934 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5740755}
I20251026 02:56:39.229463 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 568) took 5742 ms. Trace:
I20251026 02:56:39.229604 1656 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (table default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5]). Expected version 1 got 0
I20251026 02:56:39.229578 2372 rpcz_store.cc:276] 1026 02:56:33.486714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.488191 (+ 1477us) service_pool.cc:225] Handling call
1026 02:56:39.229344 (+5741153us) ts_tablet_manager.cc:1151] Deleting tablet 34c6a7c774044411a4733b115080c4c6
1026 02:56:39.229352 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.229447 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5740980}
I20251026 02:56:39.229846 2372 tablet_replica.cc:333] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:39.230029 2372 raft_consensus.cc:2243] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:39.230183 2372 raft_consensus.cc:2272] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.231052 2372 ts_tablet_manager.cc:1916] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:39.229060 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 892a5fab2af1432eb31d92d9859ea2a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 892a5fab2af1432eb31d92d9859ea2a0 already in progress: creating tablet
W20251026 02:56:39.231696 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet d0f1d6967931482e9de40b0844da6315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0f1d6967931482e9de40b0844da6315 already in progress: creating tablet
W20251026 02:56:39.231904 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 34c6a7c774044411a4733b115080c4c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34c6a7c774044411a4733b115080c4c6 already in progress: creating tablet
I20251026 02:56:39.232376 2375 tablet_bootstrap.cc:492] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:39.232556 2375 ts_tablet_manager.cc:1403] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.104s user 0.004s sys 0.001s
I20251026 02:56:39.233353 2375 raft_consensus.cc:359] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.233588 2375 raft_consensus.cc:385] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.233700 2375 raft_consensus.cc:740] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.234354 3587 raft_consensus.cc:493] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.234467 3587 raft_consensus.cc:515] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.234517 3521 consensus_queue.cc:1048] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:39.235014 3587 leader_election.cc:290] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:39.224431 2165 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:39.235544 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0de06ea2b0544402a979d9ddc487860f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.235622 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f437d8b5f5784e8fbec10d0700e90348" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:39.235859 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7da859119bd42ea9fa1290b3d4bf88c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.235939 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efd751ee4b1d433198dcd00c081f648d" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.236140 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f437d8b5f5784e8fbec10d0700e90348" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.236196 2378 tablet_bootstrap.cc:492] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:39.236371 2378 ts_tablet_manager.cc:1403] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.231s user 0.009s sys 0.017s
I20251026 02:56:39.237087 2378 raft_consensus.cc:359] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.237298 2378 raft_consensus.cc:385] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.237349 2378 raft_consensus.cc:740] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
W20251026 02:56:39.237443 2053 leader_election.cc:343] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:39.237648 2053 leader_election.cc:343] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.237721 2053 leader_election.cc:304] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.237775 1915 leader_election.cc:304] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:39.238227 3547 raft_consensus.cc:2749] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.238206 1915 leader_election.cc:304] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:39.238411 3372 raft_consensus.cc:2749] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.238540 3372 raft_consensus.cc:2749] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.238727 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 92b238ee894e4a16b3609784910266b4 (DEFAULT_TABLE table=default.Table_1f67f077a0ce4edab1c2711aa4d9556c [id=11cbf7018d894c20a980838e54ccae53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.238828 1915 leader_election.cc:304] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:39.239034 3372 raft_consensus.cc:2749] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.239360 3521 consensus_queue.cc:1048] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:39.239388 2378 consensus_queue.cc:260] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.239928 2378 ts_tablet_manager.cc:1434] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251026 02:56:39.240252 2378 tablet_bootstrap.cc:492] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:39.240958 1956 tablet_service.cc:1505] Processing CreateTablet for tablet dd7f62a7ea5344fcb1ac8499278aaec4 (DEFAULT_TABLE table=default.Table_138fc883cb304daeb0fc59f09921d7c2 [id=17090cdf8c5346b9bb0630b1beeb5e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.241083 2376 tablet_replica.cc:333] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:39.241271 2376 raft_consensus.cc:2243] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:39.241430 2376 raft_consensus.cc:2272] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.241997 2376 ts_tablet_manager.cc:1916] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.243013 2224 tablet_service.cc:1505] Processing CreateTablet for tablet dd7f62a7ea5344fcb1ac8499278aaec4 (DEFAULT_TABLE table=default.Table_138fc883cb304daeb0fc59f09921d7c2 [id=17090cdf8c5346b9bb0630b1beeb5e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.243747 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92b238ee894e4a16b3609784910266b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.244011 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7f62a7ea5344fcb1ac8499278aaec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.206717 2377 rpcz_store.cc:276] 1026 02:56:32.193213 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.194107 (+ 894us) service_pool.cc:225] Handling call
1026 02:56:39.018742 (+6824635us) ts_tablet_manager.cc:1151] Deleting tablet 63a63b796191465aab9a63d3ffb0554d
1026 02:56:39.018752 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.082870 (+ 64118us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.113797 (+ 30927us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.205874 (+ 92077us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6824493}
I20251026 02:56:39.248492 2378 tablet_bootstrap.cc:654] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:39.251631 2053 consensus_peers.cc:597] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:39.256397 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7f62a7ea5344fcb1ac8499278aaec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.260843 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 92b238ee894e4a16b3609784910266b4 (DEFAULT_TABLE table=default.Table_1f67f077a0ce4edab1c2711aa4d9556c [id=11cbf7018d894c20a980838e54ccae53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.234135 2375 consensus_queue.cc:260] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.262159 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 53264ef08ff7454faa19b18fa35bd803 (DEFAULT_TABLE table=default.Table_79f94fa4977d4066907d585cd301d523 [id=50fc2dd7098d493c816c4d379d73647f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.262595 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53264ef08ff7454faa19b18fa35bd803. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.262672 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e424a9c89e64b1c98ada17fa5c31e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.264674 2375 ts_tablet_manager.cc:1434] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.032s user 0.001s sys 0.000s
I20251026 02:56:39.265100 2375 tablet_bootstrap.cc:492] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:39.268522 2375 tablet_bootstrap.cc:654] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.268957 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 (DEFAULT_TABLE table=default.Table_10236dd7e7264f888b32afcb8c56c95f [id=5ca881edae84459c8889cbf141b18d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.269311 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.277545 2378 tablet_bootstrap.cc:492] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:39.277693 2378 ts_tablet_manager.cc:1403] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20251026 02:56:39.278618 2378 raft_consensus.cc:359] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.278918 2378 raft_consensus.cc:385] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.279016 2378 raft_consensus.cc:740] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.279474 2378 consensus_queue.cc:260] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.281159 2378 ts_tablet_manager.cc:1434] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:39.282478 3521 consensus_queue.cc:1048] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:39.282459 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 53264ef08ff7454faa19b18fa35bd803 (DEFAULT_TABLE table=default.Table_79f94fa4977d4066907d585cd301d523 [id=50fc2dd7098d493c816c4d379d73647f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.283318 2372 ts_tablet_manager.cc:1929] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:39.283789 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 7a45d350980046df88c5896e1f3dffff (DEFAULT_TABLE table=default.Table_4ded23ba48e4400b9b96e1dd82638b49 [id=a0a7c40e6bc240e6b741a83c3e7ccf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.283978 2378 tablet_bootstrap.cc:492] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:39.284305 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a45d350980046df88c5896e1f3dffff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.286358 2378 tablet_bootstrap.cc:654] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.287585 2372 log.cc:1199] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/be1c0948c0ae4563b7cb644521914cb9
I20251026 02:56:39.288034 2372 ts_tablet_manager.cc:1950] T be1c0948c0ae4563b7cb644521914cb9 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:39.288911 2381 tablet_bootstrap.cc:492] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:39.289125 2381 ts_tablet_manager.cc:1403] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.000s
I20251026 02:56:39.289932 2381 raft_consensus.cc:359] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.290190 2381 raft_consensus.cc:385] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.290306 2381 raft_consensus.cc:740] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.290616 2381 consensus_queue.cc:260] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.291205 2381 ts_tablet_manager.cc:1434] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:39.288264 2376 ts_tablet_manager.cc:1929] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:39.295590 2376 log.cc:1199] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/4a5b68772a66402bbb0b7e00bf96fcc0
I20251026 02:56:39.295557 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 574) took 5764 ms. Trace:
I20251026 02:56:39.295660 2372 rpcz_store.cc:276] 1026 02:56:33.530785 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.530852 (+ 67us) service_pool.cc:225] Handling call
1026 02:56:39.229806 (+5698954us) ts_tablet_manager.cc:1151] Deleting tablet be1c0948c0ae4563b7cb644521914cb9
1026 02:56:39.229814 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.261775 (+ 31961us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.283189 (+ 21414us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.295536 (+ 12347us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5698708}
I20251026 02:56:39.296000 2376 ts_tablet_manager.cc:1950] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:39.296278 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet be1c0948c0ae4563b7cb644521914cb9 (table default.Table_d197f56404544c8cb996d53edb9559a4 [id=777761bfdc67409bb98be7cf82bfa547]) successfully deleted
I20251026 02:56:39.297060 3587 raft_consensus.cc:493] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.297137 2377 tablet_replica.cc:333] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:39.297209 3587 raft_consensus.cc:515] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.297328 2377 raft_consensus.cc:2243] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:39.297374 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92b238ee894e4a16b3609784910266b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.297508 2377 raft_consensus.cc:2272] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.297801 3587 leader_election.cc:290] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:39.298080 2377 ts_tablet_manager.cc:1916] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.299592 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbf2a1ce78154415b23ab6d0efbc1f52" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.299763 2257 raft_consensus.cc:2468] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:39.300391 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 (DEFAULT_TABLE table=default.Table_10236dd7e7264f888b32afcb8c56c95f [id=5ca881edae84459c8889cbf141b18d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.300532 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 0c60d8227eed417595ef68d44c68ebcf (DEFAULT_TABLE table=default.Table_09b68d3ebab94c55b3c90567af20db3d [id=d35f10fbbafe4c1595eaf1a17c4c091e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.300734 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.301132 2228 tablet_service.cc:1505] Processing CreateTablet for tablet a32b1a6b856c47b993a308db94dac357 (DEFAULT_TABLE table=default.Table_111996853fae40db821e68942c9fae5e [id=e58df69dfe05472a96acaedee1928294]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.301415 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a32b1a6b856c47b993a308db94dac357. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.301846 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c60d8227eed417595ef68d44c68ebcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.303478 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 7a45d350980046df88c5896e1f3dffff (DEFAULT_TABLE table=default.Table_4ded23ba48e4400b9b96e1dd82638b49 [id=a0a7c40e6bc240e6b741a83c3e7ccf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.303771 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a45d350980046df88c5896e1f3dffff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.306758 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 3131bc2948e34c25a1bbda90f8dabeb3 (DEFAULT_TABLE table=default.Table_ce57d8a514194198ad913d3eb35269b9 [id=65f99e4eec4044b9b9a12835401e111e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.307092 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3131bc2948e34c25a1bbda90f8dabeb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.311592 2227 tablet_service.cc:1505] Processing CreateTablet for tablet 6294c8954f9749f998d5ae1c856e02ff (DEFAULT_TABLE table=default.Table_fa22c9dc29814d9cb5c4bfdec072a3cf [id=a111da525efe4df2a129972120a3871a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.311780 2381 tablet_bootstrap.cc:492] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:39.311937 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6294c8954f9749f998d5ae1c856e02ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.314061 2381 tablet_bootstrap.cc:654] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.316529 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 2e0c6a79658640c59ca9d026e3c24933 (DEFAULT_TABLE table=default.Table_63579add544846d3b2a9f267e24fda8f [id=31ab101040634eefb3abc80b3b052054]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.316866 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e0c6a79658640c59ca9d026e3c24933. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.320768 2377 ts_tablet_manager.cc:1929] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:39.320883 2377 log.cc:1199] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3791cb18fe1f4937badb4859d00647f3
I20251026 02:56:39.321236 2377 ts_tablet_manager.cc:1950] T 3791cb18fe1f4937badb4859d00647f3 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:39.321967 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 545) took 6879 ms. Trace:
I20251026 02:56:39.322052 2377 rpcz_store.cc:276] 1026 02:56:32.442228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.449618 (+ 7390us) service_pool.cc:225] Handling call
1026 02:56:39.248471 (+6798853us) ts_tablet_manager.cc:1151] Deleting tablet 3791cb18fe1f4937badb4859d00647f3
1026 02:56:39.297071 (+ 48600us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.301309 (+ 4238us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.320666 (+ 19357us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.321946 (+ 1280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6798671}
I20251026 02:56:39.322445 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 554) took 6420 ms. Trace:
I20251026 02:56:39.322513 2377 rpcz_store.cc:276] 1026 02:56:32.901611 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.901773 (+ 162us) service_pool.cc:225] Handling call
1026 02:56:39.322313 (+6420540us) ts_tablet_manager.cc:1151] Deleting tablet 3b77bb6b4722449c9699b548a9ef6314
1026 02:56:39.322322 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.322428 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6420358}
I20251026 02:56:39.322829 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 553) took 6422 ms. Trace:
I20251026 02:56:39.322896 2377 rpcz_store.cc:276] 1026 02:56:32.900045 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.912912 (+ 12867us) service_pool.cc:225] Handling call
1026 02:56:39.322727 (+6409815us) ts_tablet_manager.cc:1151] Deleting tablet 262f60fdb274476abf3533b4f47688b8
1026 02:56:39.322737 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.322815 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6409523}
I20251026 02:56:39.323150 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 556) took 6356 ms. Trace:
I20251026 02:56:39.323211 2377 rpcz_store.cc:276] 1026 02:56:32.966983 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:32.975128 (+ 8145us) service_pool.cc:225] Handling call
1026 02:56:39.323063 (+6347935us) ts_tablet_manager.cc:1151] Deleting tablet 6ea364c4a5cf4f3f99d9ff6904add697
1026 02:56:39.323072 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.323136 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6347296}
I20251026 02:56:39.323478 2377 tablet_replica.cc:333] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:39.323659 2377 raft_consensus.cc:2243] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:39.323797 2377 raft_consensus.cc:2272] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.324136 2242 tablet_service.cc:1505] Processing CreateTablet for tablet c58ce24eb8f24eee900b81bacc2a1ac3 (DEFAULT_TABLE table=default.Table_bb11d644fbd64beb9783fd730f8bb577 [id=21aff80020ca401fad4685acc5a23266]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.324537 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c58ce24eb8f24eee900b81bacc2a1ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.324882 2372 tablet_replica.cc:333] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:39.328320 2375 tablet_bootstrap.cc:492] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:39.328445 2375 ts_tablet_manager.cc:1403] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.002s
I20251026 02:56:39.329169 2375 raft_consensus.cc:359] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.329319 2375 raft_consensus.cc:385] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.329356 2375 raft_consensus.cc:740] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.331245 3587 raft_consensus.cc:493] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.331359 3587 raft_consensus.cc:515] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.331898 3587 leader_election.cc:290] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:39.333071 2375 consensus_queue.cc:260] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.334700 2375 ts_tablet_manager.cc:1434] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251026 02:56:39.336596 2381 tablet_bootstrap.cc:492] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:39.336833 2381 ts_tablet_manager.cc:1403] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.002s
I20251026 02:56:39.337961 2381 raft_consensus.cc:359] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.338224 2381 raft_consensus.cc:385] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.338413 2381 raft_consensus.cc:740] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.338833 2381 consensus_queue.cc:260] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.339843 2381 ts_tablet_manager.cc:1434] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:39.341023 2381 tablet_bootstrap.cc:492] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:39.348374 2377 ts_tablet_manager.cc:1916] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.350495 2381 tablet_bootstrap.cc:654] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.359938 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53264ef08ff7454faa19b18fa35bd803. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.371206 2381 tablet_bootstrap.cc:492] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:39.372192 2381 ts_tablet_manager.cc:1403] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.006s
I20251026 02:56:39.373824 2381 raft_consensus.cc:359] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.374148 2381 raft_consensus.cc:385] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.374296 2381 raft_consensus.cc:740] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.374708 2381 consensus_queue.cc:260] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.376392 2381 ts_tablet_manager.cc:1434] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251026 02:56:39.382483 2381 tablet_bootstrap.cc:492] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:39.384151 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet c4b0933f71cb4ac7a58986d16b613e64 (table default.Table_e6b19dda39ad45baae642c27b07db42d [id=cba6d4517d244d7580e525729e871745]). Expected version 4 got 0
I20251026 02:56:39.384488 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet e9cd10092aac40068a1c7f85e514fd37 (table default.Table_fe09be2527a840b09e3e64c27965845d [id=de5272bcc29644208ddd95bdec21254e]). Expected version 1 got 0
I20251026 02:56:39.384732 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 352a284902c94f088e2e6ae006162db0 (table default.Table_6cd4ee82e9ec406f971669b7f0722677 [id=414375003cd44b5c8c9f0249b339c28a]). Expected version 1 got 0
I20251026 02:56:39.385917 2381 tablet_bootstrap.cc:654] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.389228 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet be1c0948c0ae4563b7cb644521914cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:39.297122 1947 tablet_service.cc:1505] Processing CreateTablet for tablet 0c60d8227eed417595ef68d44c68ebcf (DEFAULT_TABLE table=default.Table_09b68d3ebab94c55b3c90567af20db3d [id=d35f10fbbafe4c1595eaf1a17c4c091e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.390317 1656 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (table default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5]). Expected version 1 got 0
I20251026 02:56:39.390584 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c60d8227eed417595ef68d44c68ebcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.390658 1656 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (table default.Table_1fa437ef7f89450bb6f672a1fb491baa [id=44c37945ff9f48e8b2c07648394ac177]). Expected version 1 got 0
I20251026 02:56:39.390584 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 550) took 5717 ms. Trace:
I20251026 02:56:39.394361 3521 raft_consensus.cc:493] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.394557 3521 raft_consensus.cc:515] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.395201 3521 leader_election.cc:290] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:39.395617 2096 tablet_service.cc:1505] Processing CreateTablet for tablet f3aea1a654314c5fb6973c33a142627c (DEFAULT_TABLE table=default.Table_f2e34abc2f4148f390dd8ccfda7e4fb8 [id=135dee19d05c4fbe9a367803f21bc6c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.392343 2376 rpcz_store.cc:276] 1026 02:56:33.672726 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.672777 (+ 51us) service_pool.cc:225] Handling call
1026 02:56:39.241041 (+5568264us) ts_tablet_manager.cc:1151] Deleting tablet 4a5b68772a66402bbb0b7e00bf96fcc0
1026 02:56:39.241050 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.266276 (+ 25226us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.288146 (+ 21870us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.390566 (+102420us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5568085}
I20251026 02:56:39.395975 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9cd10092aac40068a1c7f85e514fd37" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:39.396122 2376 tablet_replica.cc:333] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:39.297900 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 3131bc2948e34c25a1bbda90f8dabeb3 (DEFAULT_TABLE table=default.Table_ce57d8a514194198ad913d3eb35269b9 [id=65f99e4eec4044b9b9a12835401e111e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:39.396661 2194 leader_election.cc:343] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.397958 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 7bef60e3a10843b4be2022ab4b64be68 (DEFAULT_TABLE table=default.Table_3788e42710a0457c80f6ae601866b2ee [id=edb0d561b44848b0976b3a7d0d52b41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.398073 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 7053d454def04264a956aa75ceaad5c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 63eac91f5cb540e6b999ac816ee0b89d) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.399411 2053 leader_election.cc:304] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:39.399376 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 5eda85f2d53a43d68907c467c7da9bd6 (DEFAULT_TABLE table=default.Table_517fcc0e025a4b8f9ec1a392735037a9 [id=89a2069d105b44339fe06023cf322fa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.400009 3587 raft_consensus.cc:2804] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:39.400087 3587 raft_consensus.cc:493] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:39.400153 3587 raft_consensus.cc:3060] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.400111 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 28f529b6c18d412e8caa10753eab4e01 (DEFAULT_TABLE table=default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.400620 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet b819b43894574dc9ad52b4081e46628e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.396333 2376 raft_consensus.cc:2243] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:39.401425 2376 raft_consensus.cc:2272] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.401961 2376 ts_tablet_manager.cc:1916] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.401971 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9cd10092aac40068a1c7f85e514fd37" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:39.402141 2116 raft_consensus.cc:2468] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:39.301280 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 11f4c327caa644d89098496bc324b976 (DEFAULT_TABLE table=default.Table_914472564539427caff89590b14d9d01 [id=fff9b966161444b990693ad4adbfcbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.304489 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 7bef60e3a10843b4be2022ab4b64be68 (DEFAULT_TABLE table=default.Table_3788e42710a0457c80f6ae601866b2ee [id=edb0d561b44848b0976b3a7d0d52b41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.304665 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet b819b43894574dc9ad52b4081e46628e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.304785 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 28f529b6c18d412e8caa10753eab4e01 (DEFAULT_TABLE table=default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.305035 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 5eda85f2d53a43d68907c467c7da9bd6 (DEFAULT_TABLE table=default.Table_517fcc0e025a4b8f9ec1a392735037a9 [id=89a2069d105b44339fe06023cf322fa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.402769 3587 raft_consensus.cc:515] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.305295 1954 tablet_service.cc:1505] Processing CreateTablet for tablet c58ce24eb8f24eee900b81bacc2a1ac3 (DEFAULT_TABLE table=default.Table_bb11d644fbd64beb9783fd730f8bb577 [id=21aff80020ca401fad4685acc5a23266]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.316529 1945 tablet_service.cc:1505] Processing CreateTablet for tablet a32b1a6b856c47b993a308db94dac357 (DEFAULT_TABLE table=default.Table_111996853fae40db821e68942c9fae5e [id=e58df69dfe05472a96acaedee1928294]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.316844 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 2e0c6a79658640c59ca9d026e3c24933 (DEFAULT_TABLE table=default.Table_63579add544846d3b2a9f267e24fda8f [id=31ab101040634eefb3abc80b3b052054]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.317055 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbf2a1ce78154415b23ab6d0efbc1f52" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:39.332626 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b0933f71cb4ac7a58986d16b613e64" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:39.336030 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 7053d454def04264a956aa75ceaad5c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 63eac91f5cb540e6b999ac816ee0b89d) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.396497 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3131bc2948e34c25a1bbda90f8dabeb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.404224 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f529b6c18d412e8caa10753eab4e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.404378 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c58ce24eb8f24eee900b81bacc2a1ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.403301 3587 leader_election.cc:290] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:39.404208 2053 leader_election.cc:343] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:39.404942 2053 leader_election.cc:343] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.408392 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbf2a1ce78154415b23ab6d0efbc1f52" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:39.408453 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bef60e3a10843b4be2022ab4b64be68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.408906 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f4c327caa644d89098496bc324b976. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:39.408906 2053 leader_election.cc:343] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.409232 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eda85f2d53a43d68907c467c7da9bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.416422 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e0c6a79658640c59ca9d026e3c24933. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.416996 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a32b1a6b856c47b993a308db94dac357. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.417078 2378 tablet_bootstrap.cc:492] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:39.417199 2378 ts_tablet_manager.cc:1403] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.133s user 0.002s sys 0.001s
I20251026 02:56:39.417896 2378 raft_consensus.cc:359] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.419397 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eda85f2d53a43d68907c467c7da9bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.421353 3587 raft_consensus.cc:493] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.421540 3587 raft_consensus.cc:515] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.422914 3587 leader_election.cc:290] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:39.424471 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bef60e3a10843b4be2022ab4b64be68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.427891 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f529b6c18d412e8caa10753eab4e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.432526 2377 ts_tablet_manager.cc:1929] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:39.435741 2377 log.cc:1199] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/0de06ea2b0544402a979d9ddc487860f
I20251026 02:56:39.424729 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae95654d129441e08fcb3f3b355ffc22" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:39.436355 2378 raft_consensus.cc:385] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.436573 2378 raft_consensus.cc:740] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.436738 2377 ts_tablet_manager.cc:1950] T 0de06ea2b0544402a979d9ddc487860f P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:39.437009 2378 consensus_queue.cc:260] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:39.437072 2053 leader_election.cc:343] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.442394 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 559) took 6386 ms. Trace:
I20251026 02:56:39.300822 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 6294c8954f9749f998d5ae1c856e02ff (DEFAULT_TABLE table=default.Table_fa22c9dc29814d9cb5c4bfdec072a3cf [id=a111da525efe4df2a129972120a3871a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.443387 2377 rpcz_store.cc:276] 1026 02:56:33.056107 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.056164 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:39.323417 (+6267253us) ts_tablet_manager.cc:1151] Deleting tablet 0de06ea2b0544402a979d9ddc487860f
1026 02:56:39.323425 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.380685 (+ 57260us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.432439 (+ 51754us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.442357 (+ 9918us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6266936}
I20251026 02:56:39.444101 2377 tablet_replica.cc:333] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:39.444408 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6294c8954f9749f998d5ae1c856e02ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.445210 2378 ts_tablet_manager.cc:1434] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20251026 02:56:39.445472 2378 tablet_bootstrap.cc:492] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:39.446820 2378 tablet_bootstrap.cc:654] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.452219 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 11f4c327caa644d89098496bc324b976 (DEFAULT_TABLE table=default.Table_914472564539427caff89590b14d9d01 [id=fff9b966161444b990693ad4adbfcbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.452813 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f4c327caa644d89098496bc324b976. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.452418 2377 raft_consensus.cc:2243] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:39.454301 2377 raft_consensus.cc:2272] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.454962 2377 ts_tablet_manager.cc:1916] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.457451 2381 tablet_bootstrap.cc:492] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:39.457953 2381 ts_tablet_manager.cc:1403] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.009s
I20251026 02:56:39.462446 2381 raft_consensus.cc:359] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.464035 2381 raft_consensus.cc:385] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.464537 2381 raft_consensus.cc:740] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.341446 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 63a63b796191465aab9a63d3ffb0554d (table default.Table_b389bf6a8e5d4ca39fb2c59f8103652c [id=d27916c5ae83463e868d14a5d22ae281]) successfully deleted
I20251026 02:56:39.471067 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b0933f71cb4ac7a58986d16b613e64" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.471271 2257 raft_consensus.cc:2468] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:39.471915 2375 tablet_bootstrap.cc:492] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:39.473527 2375 tablet_bootstrap.cc:654] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.474906 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3aea1a654314c5fb6973c33a142627c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.477643 2117 raft_consensus.cc:1275] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:39.480760 2053 leader_election.cc:304] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.481441 3547 raft_consensus.cc:2804] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:39.481527 3547 raft_consensus.cc:493] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:39.481599 3547 raft_consensus.cc:3060] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.484357 3587 consensus_queue.cc:1048] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:39.488710 2378 tablet_bootstrap.cc:492] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:39.488852 2378 ts_tablet_manager.cc:1403] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.002s
I20251026 02:56:39.489814 2378 raft_consensus.cc:359] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.490093 2378 raft_consensus.cc:385] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.490200 2378 raft_consensus.cc:740] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.490600 2378 consensus_queue.cc:260] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.490744 2376 ts_tablet_manager.cc:1929] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:39.491282 2378 ts_tablet_manager.cc:1434] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:39.491500 2376 log.cc:1199] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/358e805370ef410e9d65ee92a8d04f34
I20251026 02:56:39.491659 2378 tablet_bootstrap.cc:492] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:39.493257 2378 tablet_bootstrap.cc:654] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.493332 2376 ts_tablet_manager.cc:1950] T 358e805370ef410e9d65ee92a8d04f34 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:39.495388 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 561) took 5632 ms. Trace:
I20251026 02:56:39.495527 2376 rpcz_store.cc:276] 1026 02:56:33.863354 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.382782 (+519428us) service_pool.cc:225] Handling call
1026 02:56:39.396077 (+5013295us) ts_tablet_manager.cc:1151] Deleting tablet 358e805370ef410e9d65ee92a8d04f34
1026 02:56:39.396087 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.450370 (+ 54283us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.490092 (+ 39722us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.495344 (+ 5252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5013099}
I20251026 02:56:39.501080 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 562) took 5637 ms. Trace:
I20251026 02:56:39.501624 2376 rpcz_store.cc:276] 1026 02:56:33.863417 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.382989 (+519572us) service_pool.cc:225] Handling call
1026 02:56:39.500499 (+5117510us) ts_tablet_manager.cc:1151] Deleting tablet 9b0406ceb60b48969ca84f6268b43dbb
1026 02:56:39.500729 (+ 230us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.501062 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5117341}
I20251026 02:56:39.503427 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 569) took 5563 ms. Trace:
I20251026 02:56:39.503819 2376 rpcz_store.cc:276] 1026 02:56:33.940002 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.397961 (+457959us) service_pool.cc:225] Handling call
1026 02:56:39.502850 (+5104889us) ts_tablet_manager.cc:1151] Deleting tablet 82e4bc6195484074b0da610a8d1b65bc
1026 02:56:39.502859 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.503407 (+ 548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5104651}
I20251026 02:56:39.506014 2376 tablet_replica.cc:333] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:39.506513 3547 raft_consensus.cc:515] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.507153 3547 leader_election.cc:290] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:39.507845 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b0933f71cb4ac7a58986d16b613e64" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:39.507691 2376 raft_consensus.cc:2243] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:39.508966 3581 raft_consensus.cc:493] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.509083 3581 raft_consensus.cc:515] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.474695 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbf2a1ce78154415b23ab6d0efbc1f52" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:39.509573 2257 raft_consensus.cc:3060] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.509703 3581 leader_election.cc:290] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:39.510239 2053 leader_election.cc:343] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:39.509564 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 127102us
I20251026 02:56:39.510546 2376 raft_consensus.cc:2272] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.511453 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 4e1291f47f384ff38eadd16040db65b8 (DEFAULT_TABLE table=default.Table_40efa920a1f94812ab827515fddc0033 [id=6c01015c58124f21b9d145102dbf8579]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.511817 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e1291f47f384ff38eadd16040db65b8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:39.512976 1916 leader_election.cc:343] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 1d7ffdaeec2240dcbe4cafdb96073a7d
I20251026 02:56:39.512197 2257 raft_consensus.cc:2468] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
W20251026 02:56:39.476269 2195 consensus_peers.cc:597] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a821ed8a686c452a975acb96b5c74778. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:39.474992 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae95654d129441e08fcb3f3b355ffc22" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.513626 2255 raft_consensus.cc:2468] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:39.513859 2195 leader_election.cc:304] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.514015 3613 consensus_queue.cc:1048] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:39.514654 3239 raft_consensus.cc:2804] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:39.514720 3239 raft_consensus.cc:493] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:39.514770 3239 raft_consensus.cc:3060] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.514815 3521 consensus_queue.cc:1048] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:39.517252 3239 raft_consensus.cc:515] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.516983 2376 ts_tablet_manager.cc:1916] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.517829 3239 leader_election.cc:290] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:39.518548 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9cd10092aac40068a1c7f85e514fd37" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:39.519028 2194 leader_election.cc:343] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.519904 1656 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe (table default.Table_47994972d3e0469c9acc96120aad4e1f [id=87349d9c6c764382b828e716e7854c59]). Expected version 3 got 0
I20251026 02:56:39.522423 3517 raft_consensus.cc:493] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.471539 2381 consensus_queue.cc:260] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.523044 3517 raft_consensus.cc:515] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.523741 3517 leader_election.cc:290] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:39.524111 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d65b0d79e2e54af9810022b683c3288b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:39.524641 2194 leader_election.cc:343] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.524864 2381 ts_tablet_manager.cc:1434] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.067s user 0.002s sys 0.000s
I20251026 02:56:39.525425 2381 tablet_bootstrap.cc:492] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:39.526108 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e4a6aa2d2b9f4653b73ae643cfe689cc" } modify_external_catalogs: true
I20251026 02:56:39.526293 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e4a6aa2d2b9f4653b73ae643cfe689cc" } modify_external_catalogs: true
I20251026 02:56:39.531392 2381 tablet_bootstrap.cc:654] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.534668 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet c2d6691bb4cf4a8292b0c082e54a91a5 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:39.535157 2308 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:39.536378 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1ae4398bf61e405697bf484105183722" } modify_external_catalogs: true
I20251026 02:56:39.536592 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e66bf47ad1824afeafd5c77400d56044" } modify_external_catalogs: true
I20251026 02:56:39.536712 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e66bf47ad1824afeafd5c77400d56044" } modify_external_catalogs: true
I20251026 02:56:39.539350 1978 raft_consensus.cc:3060] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:39.539794 1650 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 8e3cdb1f81214337a7ccc7816755f58c (table default.Table_ac84ba7eb22a4275b759937b21a2a06c [id=497a284ac16f4f6e9fcad5fe49dcbe13], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8e3cdb1f81214337a7ccc7816755f58c consensus state has no leader
I20251026 02:56:39.544641 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8248296e6b4a45d497271230d96ac519" } modify_external_catalogs: true
I20251026 02:56:39.544893 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8248296e6b4a45d497271230d96ac519" } modify_external_catalogs: true
I20251026 02:56:39.552345 3581 raft_consensus.cc:493] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.552493 3581 raft_consensus.cc:515] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.553067 3581 leader_election.cc:290] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:39.553993 1916 leader_election.cc:343] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 122de8616b8f4fcf88971d5738c47e82
I20251026 02:56:39.560297 2378 tablet_bootstrap.cc:492] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:39.560518 2378 ts_tablet_manager.cc:1403] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.000s
I20251026 02:56:39.561230 2378 raft_consensus.cc:359] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.561472 2378 raft_consensus.cc:385] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.561605 2378 raft_consensus.cc:740] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.562036 2378 consensus_queue.cc:260] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.562674 2378 ts_tablet_manager.cc:1434] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251026 02:56:39.562788 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 48372.4us
I20251026 02:56:39.563014 2378 tablet_bootstrap.cc:492] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:39.563141 3581 raft_consensus.cc:493] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.563237 3581 raft_consensus.cc:515] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.563791 3581 leader_election.cc:290] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:39.564736 3521 raft_consensus.cc:493] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:39.564754 1916 leader_election.cc:343] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 3327d26a2c2c4d0b86d98151f5940bd7
I20251026 02:56:39.564860 3521 raft_consensus.cc:515] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.565397 2053 leader_election.cc:304] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.565472 3521 leader_election.cc:290] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:39.565759 2053 leader_election.cc:304] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.565982 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa0aa6cdd0de4b0a91f553c2341e9c01" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:39.566071 3521 raft_consensus.cc:493] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.566145 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d65b0d79e2e54af9810022b683c3288b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:39.566215 3587 raft_consensus.cc:2804] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:39.566172 3521 raft_consensus.cc:515] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.566293 2111 raft_consensus.cc:2468] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:39.566593 3547 raft_consensus.cc:2804] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:39.566656 3547 raft_consensus.cc:493] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:39.566659 3521 leader_election.cc:290] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:39.566709 3547 raft_consensus.cc:3060] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:39.567041 2194 leader_election.cc:343] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.567214 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9cd10092aac40068a1c7f85e514fd37" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:39.567341 2120 raft_consensus.cc:3060] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.567473 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6108219254bd448c852096c0c967384b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:39.567862 2194 leader_election.cc:343] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.568495 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3327d26a2c2c4d0b86d98151f5940bd7" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.568697 2256 raft_consensus.cc:2393] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 0b6b0a6c4e144982b67980118cddba48 in this term.
I20251026 02:56:39.569430 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b0933f71cb4ac7a58986d16b613e64" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:39.569458 1915 leader_election.cc:304] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:39.569571 2260 raft_consensus.cc:3060] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.569722 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7ffdaeec2240dcbe4cafdb96073a7d" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.569862 3372 raft_consensus.cc:493] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.569979 3372 raft_consensus.cc:515] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.570068 3547 raft_consensus.cc:515] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.570664 3372 leader_election.cc:290] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:39.571244 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce922040298d4952af13652eeda7a234" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:39.569860 3581 raft_consensus.cc:3060] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.571982 3547 leader_election.cc:290] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:39.572296 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 049a915b35224cdaa38df77c734a0f01 (table default.Table_1818605e0bc548938337de5d5fc7a781 [id=f1f9d2ad39b34abbbe989bc3fac62b6c]) successfully deleted
I20251026 02:56:39.572330 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae95654d129441e08fcb3f3b355ffc22" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:39.572790 2260 raft_consensus.cc:2468] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:39.572890 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 4a5b68772a66402bbb0b7e00bf96fcc0 (table default.Table_1e459a892e604d56aeb212131b5a624f [id=8f39b9747e16469eb211e44a69a276e9]) successfully deleted
I20251026 02:56:39.573062 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6108219254bd448c852096c0c967384b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:39.573222 2111 raft_consensus.cc:2468] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:39.573437 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa0aa6cdd0de4b0a91f553c2341e9c01" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:39.573570 2116 raft_consensus.cc:2468] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:39.566296 3587 raft_consensus.cc:697] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:39.573973 3587 consensus_queue.cc:237] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.574795 3581 raft_consensus.cc:2749] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.575563 1915 leader_election.cc:304] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:39.576107 3581 raft_consensus.cc:2749] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:39.576602 1915 leader_election.cc:343] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 122de8616b8f4fcf88971d5738c47e82
I20251026 02:56:39.576723 1915 leader_election.cc:304] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:39.577157 3581 raft_consensus.cc:2749] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.577828 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce922040298d4952af13652eeda7a234" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.579720 2120 raft_consensus.cc:2468] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:39.579748 2194 consensus_peers.cc:597] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:39.579769 2053 leader_election.cc:304] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.580118 2195 leader_election.cc:304] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.580410 3521 raft_consensus.cc:2804] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:39.580366 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1ae4398bf61e405697bf484105183722" } modify_external_catalogs: true
I20251026 02:56:39.580488 3521 raft_consensus.cc:493] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:39.580554 3521 raft_consensus.cc:3060] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.582105 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 358e805370ef410e9d65ee92a8d04f34 (table default.Table_71583d73a49142ae9fec31193fb53542 [id=e86d9fc892a8409590c555e25b27a12f]) successfully deleted
W20251026 02:56:39.582443 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 9b0406ceb60b48969ca84f6268b43dbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b0406ceb60b48969ca84f6268b43dbb already in progress: creating tablet
W20251026 02:56:39.582677 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 82e4bc6195484074b0da610a8d1b65bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82e4bc6195484074b0da610a8d1b65bc already in progress: creating tablet
I20251026 02:56:39.583659 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet cfc3aed64ad0439cb22fd6815c973d87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b77bb6b4722449c9699b548a9ef6314) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.584081 3521 raft_consensus.cc:515] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.585696 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet a378e913112c443db869016e3b98fd0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.586635 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 3327d26a2c2c4d0b86d98151f5940bd7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7177931353164b2aa82c00b2807ccd63) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.588449 3510 raft_consensus.cc:2804] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:39.588609 3510 raft_consensus.cc:697] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:39.588860 3510 consensus_queue.cc:237] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:39.589272 2195 consensus_peers.cc:597] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d7ffdaeec2240dcbe4cafdb96073a7d. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:39.589498 2195 leader_election.cc:304] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.590260 3613 raft_consensus.cc:2804] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:39.590348 3613 raft_consensus.cc:493] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:39.590421 3613 raft_consensus.cc:3060] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.590539 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet bfa68f2c942c43e2a02e9d333bdfa2a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 262f60fdb274476abf3533b4f47688b8) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:39.590643 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e38f282e8414b7db2a6d5f4a8933e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e0b929f603b4c25b5b971ee9db683e8) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:39.593694 3613 raft_consensus.cc:515] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.594771 3613 leader_election.cc:290] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:39.594722 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6108219254bd448c852096c0c967384b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:39.595211 2194 leader_election.cc:343] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:39.580137 2053 leader_election.cc:343] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.601271 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet ce922040298d4952af13652eeda7a234 with delete_type TABLET_DATA_DELETED (replaced by tablet e830f1aa1270420c9b28a0491a1a0960) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.602200 1948 tablet_service.cc:1505] Processing CreateTablet for tablet f3aea1a654314c5fb6973c33a142627c (DEFAULT_TABLE table=default.Table_f2e34abc2f4148f390dd8ccfda7e4fb8 [id=135dee19d05c4fbe9a367803f21bc6c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.603286 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 4e1291f47f384ff38eadd16040db65b8 (DEFAULT_TABLE table=default.Table_40efa920a1f94812ab827515fddc0033 [id=6c01015c58124f21b9d145102dbf8579]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.604321 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 57dd64a73bb34ca1a46310639d5751be (DEFAULT_TABLE table=default.Table_2fab98cc17b5470a934ece6a4120a664 [id=7bb6e884c2ab43ef822540aaa7a90f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.606240 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 112181b9207e448f959f8311ad95a02b
I20251026 02:56:39.607678 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 200af441a3d24c3c8260acfa73cfe52c (DEFAULT_TABLE table=default.Table_fb45779e61d94a69ab237d499fdaea52 [id=412dd682639646c997a508c3762b7928]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.608108 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 200af441a3d24c3c8260acfa73cfe52c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.608441 1946 tablet_service.cc:1505] Processing CreateTablet for tablet d30cac4c1c684776a1d57415a5a2fcca (DEFAULT_TABLE table=default.Table_3d11f056c2674f46a40d4fe0e8473326 [id=a6751b37b93143eb86a3899eed575653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.609129 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 47037db3602246a2ab6224557286162f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.609335 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3aea1a654314c5fb6973c33a142627c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.609517 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e1291f47f384ff38eadd16040db65b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.609720 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30cac4c1c684776a1d57415a5a2fcca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.610152 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57dd64a73bb34ca1a46310639d5751be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.610116 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 200af441a3d24c3c8260acfa73cfe52c (DEFAULT_TABLE table=default.Table_fb45779e61d94a69ab237d499fdaea52 [id=412dd682639646c997a508c3762b7928]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.611114 2195 leader_election.cc:304] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.611523 2195 leader_election.cc:304] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.611742 3517 raft_consensus.cc:2804] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:39.611802 3517 raft_consensus.cc:493] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:39.611853 3517 raft_consensus.cc:3060] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.612334 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 112181b9207e448f959f8311ad95a02b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:39.614563 3517 raft_consensus.cc:515] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.615598 3517 leader_election.cc:290] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:39.615561 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa0aa6cdd0de4b0a91f553c2341e9c01" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:39.616021 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 200af441a3d24c3c8260acfa73cfe52c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:39.616022 2194 leader_election.cc:343] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.616155 3517 raft_consensus.cc:2804] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:39.616374 3517 raft_consensus.cc:697] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:39.616596 3517 consensus_queue.cc:237] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.621479 1915 leader_election.cc:304] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:39.621940 3581 raft_consensus.cc:2749] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.622778 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa0aa6cdd0de4b0a91f553c2341e9c01" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:39.622954 2116 raft_consensus.cc:3060] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.624302 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet dbf2a1ce78154415b23ab6d0efbc1f52 with delete_type TABLET_DATA_DELETED (replaced by tablet f51f8fef342b46abafff5bf12dba7184) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.624619 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet d65b0d79e2e54af9810022b683c3288b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.624711 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7a07c4cc0b48429aad305ac537734fd5" } modify_external_catalogs: true
I20251026 02:56:39.624912 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7a07c4cc0b48429aad305ac537734fd5" } modify_external_catalogs: true
I20251026 02:56:39.625089 2228 tablet_service.cc:1505] Processing CreateTablet for tablet f3aea1a654314c5fb6973c33a142627c (DEFAULT_TABLE table=default.Table_f2e34abc2f4148f390dd8ccfda7e4fb8 [id=135dee19d05c4fbe9a367803f21bc6c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.625167 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae95654d129441e08fcb3f3b355ffc22" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:39.625324 2260 raft_consensus.cc:3060] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.625468 2116 raft_consensus.cc:2468] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:39.625783 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 4e1291f47f384ff38eadd16040db65b8 (DEFAULT_TABLE table=default.Table_40efa920a1f94812ab827515fddc0033 [id=6c01015c58124f21b9d145102dbf8579]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.626111 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6108219254bd448c852096c0c967384b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:39.626174 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet c2d6691bb4cf4a8292b0c082e54a91a5
I20251026 02:56:39.626274 2120 raft_consensus.cc:3060] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.626634 1660 catalog_manager.cc:6719] ReplaceTablet: tablet c2d6691bb4cf4a8292b0c082e54a91a5 deleted and replaced by tablet 9d55e3a643d744e6b2d4485543d4b626
I20251026 02:56:39.626801 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7961528688cb41e5be1ba380f954f66a
I20251026 02:56:39.627110 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 57dd64a73bb34ca1a46310639d5751be (DEFAULT_TABLE table=default.Table_2fab98cc17b5470a934ece6a4120a664 [id=7bb6e884c2ab43ef822540aaa7a90f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.627645 2231 tablet_service.cc:1505] Processing CreateTablet for tablet d30cac4c1c684776a1d57415a5a2fcca (DEFAULT_TABLE table=default.Table_3d11f056c2674f46a40d4fe0e8473326 [id=a6751b37b93143eb86a3899eed575653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.627673 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 7961528688cb41e5be1ba380f954f66a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:39.627918 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet ae95654d129441e08fcb3f3b355ffc22 with delete_type TABLET_DATA_DELETED (replaced by tablet c2d6691bb4cf4a8292b0c082e54a91a5) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.628104 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b7d85a2941545ceaa16b22e8328b4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.629215 2120 raft_consensus.cc:2468] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:39.629310 2260 raft_consensus.cc:2468] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:39.629891 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 08c0581079674c07a694b72d6cdb3420 (DEFAULT_TABLE table=default.Table_922fb2f689694180b885eb3a1528cb24 [id=9fc86f2e6cab4fa192449bd4a4f2d1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.630268 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08c0581079674c07a694b72d6cdb3420. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.630177 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 57dd64a73bb34ca1a46310639d5751be (DEFAULT_TABLE table=default.Table_2fab98cc17b5470a934ece6a4120a664 [id=7bb6e884c2ab43ef822540aaa7a90f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.630524 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57dd64a73bb34ca1a46310639d5751be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.632586 2195 leader_election.cc:304] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.632817 3622 raft_consensus.cc:2804] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:39.632897 3622 raft_consensus.cc:697] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:39.633100 3622 consensus_queue.cc:237] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.634975 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3791cb18fe1f4937badb4859d00647f3 (table default.Table_9fb0008346da44318ccc90cd69474750 [id=c7a6636ca94a4863a89fbef09c339bc1]) successfully deleted
I20251026 02:56:39.635452 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet c2d6691bb4cf4a8292b0c082e54a91a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d55e3a643d744e6b2d4485543d4b626) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:39.635751 2195 leader_election.cc:304] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.635854 2096 tablet_service.cc:1505] Processing CreateTablet for tablet d30cac4c1c684776a1d57415a5a2fcca (DEFAULT_TABLE table=default.Table_3d11f056c2674f46a40d4fe0e8473326 [id=a6751b37b93143eb86a3899eed575653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.636253 2053 leader_election.cc:304] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.636296 3517 raft_consensus.cc:2804] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:39.636492 3517 raft_consensus.cc:697] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:39.636729 3517 consensus_queue.cc:237] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.638123 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30cac4c1c684776a1d57415a5a2fcca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.638206 2378 tablet_bootstrap.cc:654] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.644721 3547 raft_consensus.cc:2804] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:39.651895 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3aea1a654314c5fb6973c33a142627c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.652344 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57dd64a73bb34ca1a46310639d5751be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.652839 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30cac4c1c684776a1d57415a5a2fcca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.653292 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e1291f47f384ff38eadd16040db65b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.653560 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 200af441a3d24c3c8260acfa73cfe52c (DEFAULT_TABLE table=default.Table_fb45779e61d94a69ab237d499fdaea52 [id=412dd682639646c997a508c3762b7928]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.653941 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 200af441a3d24c3c8260acfa73cfe52c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.653952 3613 consensus_queue.cc:1048] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
W20251026 02:56:39.668287 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3b77bb6b4722449c9699b548a9ef6314 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b77bb6b4722449c9699b548a9ef6314 already in progress: creating tablet
W20251026 02:56:39.668810 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 262f60fdb274476abf3533b4f47688b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 262f60fdb274476abf3533b4f47688b8 already in progress: creating tablet
W20251026 02:56:39.669705 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 6ea364c4a5cf4f3f99d9ff6904add697 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ea364c4a5cf4f3f99d9ff6904add697 already in progress: creating tablet
I20251026 02:56:39.669832 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 144b957c69e64085a6f78050dea426b4
I20251026 02:56:39.670780 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 0de06ea2b0544402a979d9ddc487860f (table default.Table_6546379855b749b889279973db1820cd [id=b8be7adba1cd48ae979de506d892cc3d]) successfully deleted
I20251026 02:56:39.672586 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6c5a2f23ffbf41d294fbf17be375e630"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:39.673147 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c5a2f23ffbf41d294fbf17be375e630 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.675346 2376 ts_tablet_manager.cc:1929] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:39.675678 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ec94273925e94fb09f55249426bc4eaa
I20251026 02:56:39.676450 2376 log.cc:1199] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/7062c5ff91e14369ab69a2d199585c55
I20251026 02:56:39.677134 2376 ts_tablet_manager.cc:1950] T 7062c5ff91e14369ab69a2d199585c55 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:39.677759 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 567) took 5813 ms. Trace:
I20251026 02:56:39.678015 2376 rpcz_store.cc:276] 1026 02:56:33.863764 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.389135 (+525371us) service_pool.cc:225] Handling call
1026 02:56:39.505518 (+5116383us) ts_tablet_manager.cc:1151] Deleting tablet 7062c5ff91e14369ab69a2d199585c55
1026 02:56:39.505741 (+ 223us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.557105 (+ 51364us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.675184 (+118079us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.677714 (+ 2530us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5103380}
I20251026 02:56:39.678990 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 568) took 5755 ms. Trace:
I20251026 02:56:39.679170 2376 rpcz_store.cc:276] 1026 02:56:33.923281 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.390841 (+467560us) service_pool.cc:225] Handling call
1026 02:56:39.678763 (+5287922us) ts_tablet_manager.cc:1151] Deleting tablet d212170f056a44dfb15cfcb7f9fbae2f
1026 02:56:39.678796 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.678950 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5265577}
I20251026 02:56:39.679474 2381 tablet_bootstrap.cc:492] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:39.679680 2381 ts_tablet_manager.cc:1403] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.154s user 0.006s sys 0.000s
I20251026 02:56:39.680018 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 584) took 5144 ms. Trace:
I20251026 02:56:39.680402 2376 rpcz_store.cc:276] 1026 02:56:34.535824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.558146 (+ 22322us) service_pool.cc:225] Handling call
1026 02:56:39.679825 (+5121679us) ts_tablet_manager.cc:1151] Deleting tablet 5824831b8e184daf8b40543686c06467
1026 02:56:39.679858 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.680002 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5121496}
I20251026 02:56:39.681185 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 585) took 5131 ms. Trace:
I20251026 02:56:39.681349 2376 rpcz_store.cc:276] 1026 02:56:34.549791 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.558345 (+ 8554us) service_pool.cc:225] Handling call
1026 02:56:39.681006 (+5122661us) ts_tablet_manager.cc:1151] Deleting tablet 848e414f6e134761818039f7e98500ac
1026 02:56:39.681015 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.681143 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5122494}
I20251026 02:56:39.682020 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 583) took 5151 ms. Trace:
I20251026 02:56:39.682199 2376 rpcz_store.cc:276] 1026 02:56:34.530459 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.555162 (+ 24703us) service_pool.cc:225] Handling call
1026 02:56:39.681882 (+5126720us) ts_tablet_manager.cc:1151] Deleting tablet 6699b50d87964f6a89c365171253ad25
1026 02:56:39.681890 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.682007 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5109024}
I20251026 02:56:39.677800 2378 tablet_bootstrap.cc:492] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:39.683068 2378 ts_tablet_manager.cc:1403] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.120s user 0.003s sys 0.001s
I20251026 02:56:39.683768 2378 raft_consensus.cc:359] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.684005 2378 raft_consensus.cc:385] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.684098 2378 raft_consensus.cc:740] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.682842 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 588) took 5110 ms. Trace:
I20251026 02:56:39.684649 2376 rpcz_store.cc:276] 1026 02:56:34.572273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.572330 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:39.682675 (+5110345us) ts_tablet_manager.cc:1151] Deleting tablet 5ddc7972f29a429ea51f6c6c1a074baf
1026 02:56:39.682700 (+ 25us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.682826 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5071532}
I20251026 02:56:39.685192 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 600) took 5099 ms. Trace:
I20251026 02:56:39.685315 2376 rpcz_store.cc:276] 1026 02:56:34.585855 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.056836 (+470981us) service_pool.cc:225] Handling call
1026 02:56:39.685054 (+4628218us) ts_tablet_manager.cc:1151] Deleting tablet c09153fbfe18434393c188058a3461d5
1026 02:56:39.685066 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.685176 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4621482}
I20251026 02:56:39.685801 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 626) took 4567 ms. Trace:
I20251026 02:56:39.685921 2376 rpcz_store.cc:276] 1026 02:56:35.118194 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.427956 (+309762us) service_pool.cc:225] Handling call
1026 02:56:39.685713 (+4257757us) ts_tablet_manager.cc:1151] Deleting tablet a74d6a4e041347578b0976a8b54bf22c
1026 02:56:39.685722 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.685787 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4223466}
I20251026 02:56:39.686357 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 627) took 4568 ms. Trace:
I20251026 02:56:39.686514 2376 rpcz_store.cc:276] 1026 02:56:35.118294 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.433612 (+315318us) service_pool.cc:225] Handling call
1026 02:56:39.686272 (+4252660us) ts_tablet_manager.cc:1151] Deleting tablet 657e07d0b17c44d4b78b97210f17ce40
1026 02:56:39.686279 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.686343 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4221177}
I20251026 02:56:39.686944 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 628) took 4568 ms. Trace:
I20251026 02:56:39.687057 2376 rpcz_store.cc:276] 1026 02:56:35.118362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.448194 (+329832us) service_pool.cc:225] Handling call
1026 02:56:39.686859 (+4238665us) ts_tablet_manager.cc:1151] Deleting tablet 3efafaac6c634d088396d56c8debd990
1026 02:56:39.686867 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.686932 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4219103}
I20251026 02:56:39.687505 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 625) took 4609 ms. Trace:
I20251026 02:56:39.687631 2376 rpcz_store.cc:276] 1026 02:56:35.077650 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.423784 (+346134us) service_pool.cc:225] Handling call
1026 02:56:39.687418 (+4263634us) ts_tablet_manager.cc:1151] Deleting tablet bc4ca4423e3a4e0c96846babec931fec
1026 02:56:39.687426 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.687493 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4217253}
I20251026 02:56:39.688038 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 634) took 4192 ms. Trace:
I20251026 02:56:39.688165 2376 rpcz_store.cc:276] 1026 02:56:35.495740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.503156 (+ 7416us) service_pool.cc:225] Handling call
1026 02:56:39.687947 (+4184791us) ts_tablet_manager.cc:1151] Deleting tablet 902cc4f9dbbd4d8baa829448f2a4687a
1026 02:56:39.687956 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.688026 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4184447}
I20251026 02:56:39.688707 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 629) took 4570 ms. Trace:
I20251026 02:56:39.688835 2376 rpcz_store.cc:276] 1026 02:56:35.118444 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.462278 (+343834us) service_pool.cc:225] Handling call
1026 02:56:39.688625 (+4226347us) ts_tablet_manager.cc:1151] Deleting tablet 1219e8bdfde9444f9bfdf1d626389e88
1026 02:56:39.688633 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.688696 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4183935}
I20251026 02:56:39.690256 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 637) took 4067 ms. Trace:
I20251026 02:56:39.693289 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_37a214cdf9c846cf8e0f55e6cbd3d274" } new_table_name: "default.Table_c062fd73dfee44db8acf3124482a05ed" modify_external_catalogs: true
I20251026 02:56:39.695673 3521 leader_election.cc:290] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:39.696659 2381 raft_consensus.cc:359] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.696899 2381 raft_consensus.cc:385] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.696988 2381 raft_consensus.cc:740] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.698105 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 7961528688cb41e5be1ba380f954f66a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.698134 2381 consensus_queue.cc:260] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.698947 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 08c0581079674c07a694b72d6cdb3420 (DEFAULT_TABLE table=default.Table_922fb2f689694180b885eb3a1528cb24 [id=9fc86f2e6cab4fa192449bd4a4f2d1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.699350 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08c0581079674c07a694b72d6cdb3420. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.700181 3622 consensus_queue.cc:1048] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:39.700377 2237 tablet_service.cc:1505] Processing CreateTablet for tablet d4c863ef4d3c43eabb1b6acf13915963 (DEFAULT_TABLE table=default.Table_5a1e2e5e59314659a935626df27c1d18 [id=b9246d4def784f13b8357047d3f5eb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.700729 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c863ef4d3c43eabb1b6acf13915963. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.701592 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4a0e936ac9b94d1eb3da682e6bfec5c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:39.702121 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a0e936ac9b94d1eb3da682e6bfec5c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.704100 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a905e23c621542fa802270f96a8dfd8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:39.704457 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 144b957c69e64085a6f78050dea426b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.704784 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet ec94273925e94fb09f55249426bc4eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.704921 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet c2d6691bb4cf4a8292b0c082e54a91a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d55e3a643d744e6b2d4485543d4b626) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.705440 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 112181b9207e448f959f8311ad95a02b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.708472 2381 ts_tablet_manager.cc:1434] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20251026 02:56:39.709300 2381 tablet_bootstrap.cc:492] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:39.709594 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a905e23c621542fa802270f96a8dfd8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.677044 2083 tablet_service.cc:1552] Processing DeleteTablet for tablet ec94273925e94fb09f55249426bc4eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:39.710419 3547 raft_consensus.cc:697] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:39.710673 3547 consensus_queue.cc:237] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.689563 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 144b957c69e64085a6f78050dea426b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:39.695967 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d65b0d79e2e54af9810022b683c3288b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
W20251026 02:56:39.697605 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 61392.5us
I20251026 02:56:39.711001 2116 raft_consensus.cc:3060] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.704439 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 8fdadd053cc34aeca0d491eb3200207f (DEFAULT_TABLE table=default.Table_ce756cf85c594b7f97cbbfb345be2856 [id=c383ba44e6eb45c8915d3a0af171f6c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.712531 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fdadd053cc34aeca0d491eb3200207f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.716370 2372 raft_consensus.cc:2243] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:39.716531 2372 raft_consensus.cc:2272] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.717015 2372 ts_tablet_manager.cc:1916] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:39.719877 2053 consensus_peers.cc:597] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 049a915b35224cdaa38df77c734a0f01. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:39.725785 2116 raft_consensus.cc:2468] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:39.727051 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b1f6e2c58c7f4367813eb66ebda5e0ca
I20251026 02:56:39.722033 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_502d978bc87a4bdcbd6d61a2710dc639"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:39.729943 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet b1f6e2c58c7f4367813eb66ebda5e0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:39.730340 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_502d978bc87a4bdcbd6d61a2710dc639 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.731460 2195 leader_election.cc:304] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:39.732069 3613 raft_consensus.cc:2804] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:39.732192 3613 raft_consensus.cc:697] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:39.732512 3613 consensus_queue.cc:237] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.733618 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0a47e554e1874269a6d13aef19c29678" } new_table_name: "default.Table_d68a91a3695240a39e5526b9cff6f6f2" modify_external_catalogs: true
I20251026 02:56:39.704411 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 7801f073d5d94076af9299492f86fad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.735440 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3ee83bc1426c40dfb0c28e58715a9476"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:39.736253 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ee83bc1426c40dfb0c28e58715a9476 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.736461 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ef87e98153774f2b92a5b45f9191806e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:39.737339 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef87e98153774f2b92a5b45f9191806e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.693331 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 112181b9207e448f959f8311ad95a02b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.693806 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 08c0581079674c07a694b72d6cdb3420 (DEFAULT_TABLE table=default.Table_922fb2f689694180b885eb3a1528cb24 [id=9fc86f2e6cab4fa192449bd4a4f2d1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.738363 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 7062c5ff91e14369ab69a2d199585c55 (table default.Table_d48e174e391f42d6aa0b05c322b5b828 [id=91a568329da84558b31575e5127f873f]) successfully deleted
W20251026 02:56:39.738651 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet d212170f056a44dfb15cfcb7f9fbae2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d212170f056a44dfb15cfcb7f9fbae2f already in progress: creating tablet
I20251026 02:56:39.738813 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08c0581079674c07a694b72d6cdb3420. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:39.739092 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 5824831b8e184daf8b40543686c06467 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5824831b8e184daf8b40543686c06467 already in progress: creating tablet
W20251026 02:56:39.739347 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 848e414f6e134761818039f7e98500ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 848e414f6e134761818039f7e98500ac already in progress: creating tablet
I20251026 02:56:39.740809 2090 tablet_service.cc:1505] Processing CreateTablet for tablet ebb20ca32f574758bc1eb67837fbeb6a (DEFAULT_TABLE table=default.Table_038ad7af9e93421f9ab149ff87a4c7d8 [id=381d646cb0384b2e823550400137c406]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.741420 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebb20ca32f574758bc1eb67837fbeb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.695048 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet c2d6691bb4cf4a8292b0c082e54a91a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d55e3a643d744e6b2d4485543d4b626) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.695719 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d65b0d79e2e54af9810022b683c3288b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:39.699713 1943 tablet_service.cc:1505] Processing CreateTablet for tablet d4c863ef4d3c43eabb1b6acf13915963 (DEFAULT_TABLE table=default.Table_5a1e2e5e59314659a935626df27c1d18 [id=b9246d4def784f13b8357047d3f5eb54]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:39.742414 2194 leader_election.cc:343] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.742472 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c863ef4d3c43eabb1b6acf13915963. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.700271 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 144b957c69e64085a6f78050dea426b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.749876 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 0937c297edcb4210abad0431e5108adb (DEFAULT_TABLE table=default.Table_3f3634e9694e4cafab2335c0fab273cc [id=90cc9ca24d9d436fb51b1a773618556a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.750298 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0937c297edcb4210abad0431e5108adb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:39.751931 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 6699b50d87964f6a89c365171253ad25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6699b50d87964f6a89c365171253ad25 already in progress: creating tablet
W20251026 02:56:39.752287 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 5ddc7972f29a429ea51f6c6c1a074baf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ddc7972f29a429ea51f6c6c1a074baf already in progress: creating tablet
W20251026 02:56:39.752538 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet c09153fbfe18434393c188058a3461d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c09153fbfe18434393c188058a3461d5 already in progress: creating tablet
I20251026 02:56:39.702498 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.704306 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet ec94273925e94fb09f55249426bc4eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.705309 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 7961528688cb41e5be1ba380f954f66a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.711637 2378 consensus_queue.cc:260] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.754931 2027 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:39.757656 2376 rpcz_store.cc:276] 1026 02:56:35.623295 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.624476 (+ 1181us) service_pool.cc:225] Handling call
1026 02:56:39.690129 (+4065653us) ts_tablet_manager.cc:1151] Deleting tablet 1dbbfb77851d443f8cf4d798f3e07e59
1026 02:56:39.690139 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.690239 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4065330}
I20251026 02:56:39.758795 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 3edd263ba2644e078ac32440ad5df939 (DEFAULT_TABLE table=default.Table_dfc5b9165b6d446380c0e6f7d3752aa6 [id=599039c1bb7645bc97544802fb302c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.758984 2378 ts_tablet_manager.cc:1434] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.076s user 0.001s sys 0.001s
I20251026 02:56:39.759246 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3edd263ba2644e078ac32440ad5df939. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.760890 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 638) took 4089 ms. Trace:
I20251026 02:56:39.761034 2376 rpcz_store.cc:276] 1026 02:56:35.671653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.671727 (+ 74us) service_pool.cc:225] Handling call
1026 02:56:39.758218 (+4086491us) ts_tablet_manager.cc:1151] Deleting tablet 1ffead7ca2d946c78aa4b8a602865fc8
1026 02:56:39.760627 (+ 2409us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.760872 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4086278}
I20251026 02:56:39.761559 2376 tablet_replica.cc:333] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
W20251026 02:56:39.764003 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet a74d6a4e041347578b0976a8b54bf22c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a74d6a4e041347578b0976a8b54bf22c already in progress: creating tablet
W20251026 02:56:39.764557 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 657e07d0b17c44d4b78b97210f17ce40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 657e07d0b17c44d4b78b97210f17ce40 already in progress: creating tablet
W20251026 02:56:39.764830 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3efafaac6c634d088396d56c8debd990 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3efafaac6c634d088396d56c8debd990 already in progress: creating tablet
W20251026 02:56:39.765236 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet bc4ca4423e3a4e0c96846babec931fec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc4ca4423e3a4e0c96846babec931fec already in progress: creating tablet
I20251026 02:56:39.765272 3647 raft_consensus.cc:493] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.765399 3581 raft_consensus.cc:493] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.765574 2378 tablet_bootstrap.cc:492] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:39.765537 3581 raft_consensus.cc:515] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.766193 3581 leader_election.cc:290] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:39.767194 2378 tablet_bootstrap.cc:654] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:39.767519 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 902cc4f9dbbd4d8baa829448f2a4687a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 902cc4f9dbbd4d8baa829448f2a4687a already in progress: creating tablet
W20251026 02:56:39.767737 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 1219e8bdfde9444f9bfdf1d626389e88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1219e8bdfde9444f9bfdf1d626389e88 already in progress: creating tablet
W20251026 02:56:39.768095 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 1dbbfb77851d443f8cf4d798f3e07e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dbbfb77851d443f8cf4d798f3e07e59 already in progress: creating tablet
W20251026 02:56:39.768153 2194 consensus_peers.cc:597] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 358e805370ef410e9d65ee92a8d04f34. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:39.768488 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 1ffead7ca2d946c78aa4b8a602865fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ffead7ca2d946c78aa4b8a602865fc8 already in progress: creating tablet
I20251026 02:56:39.769239 1632 master_runner.cc:186] Error getting master registration for 127.1.150.60:33729: OK, instance_id { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" instance_seqno: 1761447396760524 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251026 02:56:39.769487 1632 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 02:56:39.769987 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44af74033ccf4d048475f12d164938f5" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.770056 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44af74033ccf4d048475f12d164938f5" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:39.765550 3647 raft_consensus.cc:515] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.771086 3647 leader_election.cc:290] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:39.771461 1915 leader_election.cc:304] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:39.771912 3581 raft_consensus.cc:2749] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.773322 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e38f282e8414b7db2a6d5f4a8933e84" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:39.774226 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e38f282e8414b7db2a6d5f4a8933e84" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.761806 2376 raft_consensus.cc:2243] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:39.782127 2376 raft_consensus.cc:2272] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.782785 2376 ts_tablet_manager.cc:1916] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.785610 2377 ts_tablet_manager.cc:1929] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:39.785791 2377 log.cc:1199] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/1e56b259a314422aba35eb129021bbeb
I20251026 02:56:39.786621 2377 ts_tablet_manager.cc:1950] T 1e56b259a314422aba35eb129021bbeb P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:39.787460 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 560) took 6730 ms. Trace:
I20251026 02:56:39.689163 2085 tablet_service.cc:1505] Processing CreateTablet for tablet d4c863ef4d3c43eabb1b6acf13915963 (DEFAULT_TABLE table=default.Table_5a1e2e5e59314659a935626df27c1d18 [id=b9246d4def784f13b8357047d3f5eb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.789849 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c863ef4d3c43eabb1b6acf13915963. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.790596 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 551cc0916a21463d9190d14a31e79988 (DEFAULT_TABLE table=default.Table_a263fecc382f46039c6cdad42a49517d [id=da76f3f1e547476ebeb94360c5e77ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.791075 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 551cc0916a21463d9190d14a31e79988. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.792593 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 45d5062fc24948c98c6274d2cefc04b5 (DEFAULT_TABLE table=default.Table_465e7833737a45b997d9aa79e8c1bf5e [id=948a2c4552324522b7f8f0bab9f1bc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.792970 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d5062fc24948c98c6274d2cefc04b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.807544 2376 ts_tablet_manager.cc:1929] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:39.807893 2376 log.cc:1199] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/0efdcc36ff5b47379e0a76512e05d379
I20251026 02:56:39.808590 2376 ts_tablet_manager.cc:1950] T 0efdcc36ff5b47379e0a76512e05d379 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
W20251026 02:56:39.809857 2053 leader_election.cc:343] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.810161 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 639) took 4069 ms. Trace:
W20251026 02:56:39.810601 2053 leader_election.cc:343] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.810762 2053 leader_election.cc:304] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.810554 2376 rpcz_store.cc:276] 1026 02:56:35.740771 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.744046 (+ 3275us) service_pool.cc:225] Handling call
1026 02:56:39.761515 (+4017469us) ts_tablet_manager.cc:1151] Deleting tablet 0efdcc36ff5b47379e0a76512e05d379
1026 02:56:39.761524 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.801460 (+ 39936us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.807403 (+ 5943us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.810138 (+ 2735us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4017220}
I20251026 02:56:39.811033 3631 raft_consensus.cc:2749] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.811736 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 640) took 4070 ms. Trace:
I20251026 02:56:39.811962 2376 rpcz_store.cc:276] 1026 02:56:35.740965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.744156 (+ 3191us) service_pool.cc:225] Handling call
1026 02:56:39.811456 (+4067300us) ts_tablet_manager.cc:1151] Deleting tablet 24f16e40a30c41788a6f9813c235c5ab
1026 02:56:39.811471 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.811689 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4067059}
I20251026 02:56:39.812706 2376 tablet_replica.cc:333] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:39.813045 2376 raft_consensus.cc:2243] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:39.813454 2376 raft_consensus.cc:2272] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.814499 2376 ts_tablet_manager.cc:1916] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.817564 2375 tablet_bootstrap.cc:492] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:39.817919 2375 ts_tablet_manager.cc:1403] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.346s user 0.013s sys 0.008s
I20251026 02:56:39.819020 2375 raft_consensus.cc:359] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.819320 2375 raft_consensus.cc:385] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.819413 2375 raft_consensus.cc:740] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.819649 2375 consensus_queue.cc:260] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.820281 2094 tablet_service.cc:1505] Processing CreateTablet for tablet e6bc09dc66d64d668e424c929d191c02 (DEFAULT_TABLE table=default.Table_e9c6f4d328654c33b5edf5cf73450028 [id=93c4028c87ab4d7682716bf2efa9cd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.820931 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6bc09dc66d64d668e424c929d191c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.825441 3587 consensus_queue.cc:1048] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:39.826081 3581 raft_consensus.cc:493] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.826242 3581 raft_consensus.cc:515] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.826870 3581 leader_election.cc:290] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:39.826972 2376 ts_tablet_manager.cc:1929] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251026 02:56:39.827780 1916 leader_election.cc:343] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: cfc3aed64ad0439cb22fd6815c973d87
I20251026 02:56:39.827219 2376 log.cc:1199] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/115ccccf16b74efc9f24b607cf0c74d1
W20251026 02:56:39.828374 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 58658.4us
I20251026 02:56:39.829017 2376 ts_tablet_manager.cc:1950] T 115ccccf16b74efc9f24b607cf0c74d1 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:39.829491 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 8fdadd053cc34aeca0d491eb3200207f (DEFAULT_TABLE table=default.Table_ce756cf85c594b7f97cbbfb345be2856 [id=c383ba44e6eb45c8915d3a0af171f6c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.829926 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fdadd053cc34aeca0d491eb3200207f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.831068 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 641) took 4078 ms. Trace:
I20251026 02:56:39.831386 2376 rpcz_store.cc:276] 1026 02:56:35.752850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.752929 (+ 79us) service_pool.cc:225] Handling call
1026 02:56:39.812637 (+4059708us) ts_tablet_manager.cc:1151] Deleting tablet 115ccccf16b74efc9f24b607cf0c74d1
1026 02:56:39.812646 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.820846 (+ 8200us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.826793 (+ 5947us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.831006 (+ 4213us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4059445}
I20251026 02:56:39.832698 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet b1f6e2c58c7f4367813eb66ebda5e0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.833237 3587 raft_consensus.cc:493] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.833500 3587 raft_consensus.cc:515] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.834084 3587 leader_election.cc:290] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:39.834096 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 642) took 4080 ms. Trace:
I20251026 02:56:39.834358 2376 rpcz_store.cc:276] 1026 02:56:35.754088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.754194 (+ 106us) service_pool.cc:225] Handling call
1026 02:56:39.833841 (+4079647us) ts_tablet_manager.cc:1151] Deleting tablet 351cd60b076a49949f03640c981845c4
1026 02:56:39.833882 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.834050 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4075850}
I20251026 02:56:39.835280 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 643) took 3871 ms. Trace:
I20251026 02:56:39.835593 2376 rpcz_store.cc:276] 1026 02:56:35.963887 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.963959 (+ 72us) service_pool.cc:225] Handling call
1026 02:56:39.835068 (+3871109us) ts_tablet_manager.cc:1151] Deleting tablet 30c619ad37364747b6ebf97a73ef45a2
1026 02:56:39.835081 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.835231 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3870880}
I20251026 02:56:39.836541 2376 tablet_replica.cc:333] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:39.837110 2376 raft_consensus.cc:2243] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:39.837546 2376 raft_consensus.cc:2272] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.838845 2376 ts_tablet_manager.cc:1916] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.840191 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfa68f2c942c43e2a02e9d333bdfa2a2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:39.841878 2053 leader_election.cc:343] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.845917 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 51a39824f5404d45b1756a2581aa1da4 (DEFAULT_TABLE table=default.Table_e9df6f46f0f3408d9aa07452b0d7e77d [id=6b4a332c932b4b1c80e72e86274dc079]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.846405 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51a39824f5404d45b1756a2581aa1da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.827441 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfc3aed64ad0439cb22fd6815c973d87" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.848255 2260 raft_consensus.cc:2393] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 0b6b0a6c4e144982b67980118cddba48 in this term.
I20251026 02:56:39.851205 1915 leader_election.cc:304] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:39.851342 1943 tablet_service.cc:1505] Processing CreateTablet for tablet ebb20ca32f574758bc1eb67837fbeb6a (DEFAULT_TABLE table=default.Table_038ad7af9e93421f9ab149ff87a4c7d8 [id=381d646cb0384b2e823550400137c406]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.851730 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebb20ca32f574758bc1eb67837fbeb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.851850 3581 raft_consensus.cc:3060] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.855118 3581 raft_consensus.cc:2749] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.855842 3639 tablet.cc:1722] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:39.848155 2377 rpcz_store.cc:276] 1026 02:56:33.056779 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.056826 (+ 47us) service_pool.cc:225] Handling call
1026 02:56:39.444045 (+6387219us) ts_tablet_manager.cc:1151] Deleting tablet 1e56b259a314422aba35eb129021bbeb
1026 02:56:39.444058 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.708341 (+264283us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.785504 (+ 77163us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.787439 (+ 1935us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6387010}
I20251026 02:56:39.856297 2237 tablet_service.cc:1505] Processing CreateTablet for tablet ebb20ca32f574758bc1eb67837fbeb6a (DEFAULT_TABLE table=default.Table_038ad7af9e93421f9ab149ff87a4c7d8 [id=381d646cb0384b2e823550400137c406]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.856712 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebb20ca32f574758bc1eb67837fbeb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.856917 2377 tablet_replica.cc:333] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:39.857138 2377 raft_consensus.cc:2243] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:39.857518 2377 raft_consensus.cc:2272] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.858088 2377 ts_tablet_manager.cc:1916] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:39.863716 2378 tablet_bootstrap.cc:492] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:39.863888 2378 ts_tablet_manager.cc:1403] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.003s
I20251026 02:56:39.864662 2378 raft_consensus.cc:359] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.864854 2378 raft_consensus.cc:385] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:39.864907 2378 raft_consensus.cc:740] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:39.865072 2378 consensus_queue.cc:260] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:39.865536 2378 ts_tablet_manager.cc:1434] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:39.865788 2378 tablet_bootstrap.cc:492] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:39.866847 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 0efdcc36ff5b47379e0a76512e05d379 (table default.Table_1972d81a4c494b7c97894d4991fed574 [id=d9f8b0fbb3ed416fb380a5717fb29dba]) successfully deleted
W20251026 02:56:39.867219 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 24f16e40a30c41788a6f9813c235c5ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24f16e40a30c41788a6f9813c235c5ab already in progress: creating tablet
I20251026 02:56:39.867434 2378 tablet_bootstrap.cc:654] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.867560 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 115ccccf16b74efc9f24b607cf0c74d1 (table default.Table_7b6687267a1a4b2da7e02949e7b4c081 [id=ad981d3883cc4ee5b77defff9651ef31]) successfully deleted
W20251026 02:56:39.867812 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 351cd60b076a49949f03640c981845c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 351cd60b076a49949f03640c981845c4 already in progress: creating tablet
W20251026 02:56:39.868078 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 30c619ad37364747b6ebf97a73ef45a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30c619ad37364747b6ebf97a73ef45a2 already in progress: creating tablet
I20251026 02:56:39.869612 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 17d2cc6b16d34331bb80c576c0cd2ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:39.869942 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 45d5062fc24948c98c6274d2cefc04b5 (DEFAULT_TABLE table=default.Table_465e7833737a45b997d9aa79e8c1bf5e [id=948a2c4552324522b7f8f0bab9f1bc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.870231 1954 tablet_service.cc:1505] Processing CreateTablet for tablet e6bc09dc66d64d668e424c929d191c02 (DEFAULT_TABLE table=default.Table_e9c6f4d328654c33b5edf5cf73450028 [id=93c4028c87ab4d7682716bf2efa9cd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.870337 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 0937c297edcb4210abad0431e5108adb (DEFAULT_TABLE table=default.Table_3f3634e9694e4cafab2335c0fab273cc [id=90cc9ca24d9d436fb51b1a773618556a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.870537 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d5062fc24948c98c6274d2cefc04b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.827759 2381 tablet_bootstrap.cc:654] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.870604 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 3edd263ba2644e078ac32440ad5df939 (DEFAULT_TABLE table=default.Table_dfc5b9165b6d446380c0e6f7d3752aa6 [id=599039c1bb7645bc97544802fb302c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.870740 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 3edd263ba2644e078ac32440ad5df939 (DEFAULT_TABLE table=default.Table_dfc5b9165b6d446380c0e6f7d3752aa6 [id=599039c1bb7645bc97544802fb302c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.870929 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3edd263ba2644e078ac32440ad5df939. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.871140 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 551cc0916a21463d9190d14a31e79988 (DEFAULT_TABLE table=default.Table_a263fecc382f46039c6cdad42a49517d [id=da76f3f1e547476ebeb94360c5e77ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.871325 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 0937c297edcb4210abad0431e5108adb (DEFAULT_TABLE table=default.Table_3f3634e9694e4cafab2335c0fab273cc [id=90cc9ca24d9d436fb51b1a773618556a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.871475 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 51a39824f5404d45b1756a2581aa1da4 (DEFAULT_TABLE table=default.Table_e9df6f46f0f3408d9aa07452b0d7e77d [id=6b4a332c932b4b1c80e72e86274dc079]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.872462 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 551cc0916a21463d9190d14a31e79988 (DEFAULT_TABLE table=default.Table_a263fecc382f46039c6cdad42a49517d [id=da76f3f1e547476ebeb94360c5e77ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.872749 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 551cc0916a21463d9190d14a31e79988. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.873383 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 51a39824f5404d45b1756a2581aa1da4 (DEFAULT_TABLE table=default.Table_e9df6f46f0f3408d9aa07452b0d7e77d [id=6b4a332c932b4b1c80e72e86274dc079]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.873744 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51a39824f5404d45b1756a2581aa1da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.832871 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 8fdadd053cc34aeca0d491eb3200207f (DEFAULT_TABLE table=default.Table_ce756cf85c594b7f97cbbfb345be2856 [id=c383ba44e6eb45c8915d3a0af171f6c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.874503 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0937c297edcb4210abad0431e5108adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.874749 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fdadd053cc34aeca0d491eb3200207f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.878322 2227 tablet_service.cc:1505] Processing CreateTablet for tablet e6bc09dc66d64d668e424c929d191c02 (DEFAULT_TABLE table=default.Table_e9c6f4d328654c33b5edf5cf73450028 [id=93c4028c87ab4d7682716bf2efa9cd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.878712 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6bc09dc66d64d668e424c929d191c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.833493 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet b1f6e2c58c7f4367813eb66ebda5e0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:39 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:39.840258 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfa68f2c942c43e2a02e9d333bdfa2a2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:39.820295 2375 ts_tablet_manager.cc:1434] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:39.882270 2375 tablet_bootstrap.cc:492] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:39.882350 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 551cc0916a21463d9190d14a31e79988. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.883129 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0937c297edcb4210abad0431e5108adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.884548 2375 tablet_bootstrap.cc:654] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:39.888480 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6bc09dc66d64d668e424c929d191c02. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:39.888564 2194 consensus_peers.cc:597] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a5b68772a66402bbb0b7e00bf96fcc0. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:39.893033 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51a39824f5404d45b1756a2581aa1da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.894858 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_51cf6f44f72b4fea9695ac3880595e80" } new_table_name: "default.Table_46eca4a0a83f4d3091c55183b402758d" modify_external_catalogs: true
I20251026 02:56:39.894794 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_27c4161c9b184347971d3035b388e746"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:39.895493 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27c4161c9b184347971d3035b388e746 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.896193 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7e7480c10911493eaab01f1e2b1a88ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:39.896719 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e7480c10911493eaab01f1e2b1a88ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.899972 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f41eab2536d54c839327b1084896bba4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:39.900542 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f41eab2536d54c839327b1084896bba4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:39.906816 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 45d5062fc24948c98c6274d2cefc04b5 (DEFAULT_TABLE table=default.Table_465e7833737a45b997d9aa79e8c1bf5e [id=948a2c4552324522b7f8f0bab9f1bc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.907297 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d5062fc24948c98c6274d2cefc04b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.907385 2372 ts_tablet_manager.cc:1929] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:39.907506 2372 log.cc:1199] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/47037db3602246a2ab6224557286162f
I20251026 02:56:39.907810 2372 ts_tablet_manager.cc:1950] T 47037db3602246a2ab6224557286162f P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:39.910722 1982 raft_consensus.cc:3060] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:39.911355 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3edd263ba2644e078ac32440ad5df939. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:39.909835 2053 leader_election.cc:343] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.912326 2053 leader_election.cc:304] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.912916 3631 raft_consensus.cc:493] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.913024 3631 raft_consensus.cc:515] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.913697 3631 leader_election.cc:290] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:39.913848 3631 raft_consensus.cc:2749] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.914837 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21b936fa4f6a4ff6af321971a30cccbe" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:39.915586 2053 leader_election.cc:343] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.916261 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21b936fa4f6a4ff6af321971a30cccbe" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:39.916704 2194 consensus_peers.cc:597] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:39.917214 2053 leader_election.cc:343] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:39.917382 2053 leader_election.cc:304] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:39.917825 3547 raft_consensus.cc:2749] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:39.917985 2085 tablet_service.cc:1505] Processing CreateTablet for tablet f502f8edc2634e9288def155335ab1e3 (DEFAULT_TABLE table=default.Table_8466ce8a41f84c3797f015d455b86d1c [id=569b458344a740a880cf0c90077a70de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.918402 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f502f8edc2634e9288def155335ab1e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:39.919409 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 51344.8us
I20251026 02:56:39.920120 1945 tablet_service.cc:1505] Processing CreateTablet for tablet f502f8edc2634e9288def155335ab1e3 (DEFAULT_TABLE table=default.Table_8466ce8a41f84c3797f015d455b86d1c [id=569b458344a740a880cf0c90077a70de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.920780 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f502f8edc2634e9288def155335ab1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.923760 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 63f1abc4477f4198ad2cc8481d69df73 (DEFAULT_TABLE table=default.Table_7ba3891cfe0b46329d460a7a3881d8d0 [id=3cd62b627c1d4b5d875e4bf0a9158aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.924232 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63f1abc4477f4198ad2cc8481d69df73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.928566 2083 tablet_service.cc:1505] Processing CreateTablet for tablet ecbed9dd56e94cad8d7387155b934c50 (DEFAULT_TABLE table=default.Table_07a34faac32549c99cb5b5a936a56945 [id=d3363ba8a35846b5bd1ae2b334e713cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.929114 3651 tablet.cc:1722] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:39.929805 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 234112c1e45948ec81f67e17639be32d (DEFAULT_TABLE table=default.Table_23d0e02f7246407f8337ddd38381fa9c [id=5b4866df0e9442f899850a0e47e90b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.931001 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 1e56b259a314422aba35eb129021bbeb (table default.Table_25607539f64440cb8ff2c24a0a3a3bdd [id=5dbfb5d912ba433baa46bd52bbf6dbee]) successfully deleted
I20251026 02:56:39.933437 1948 tablet_service.cc:1505] Processing CreateTablet for tablet ecbed9dd56e94cad8d7387155b934c50 (DEFAULT_TABLE table=default.Table_07a34faac32549c99cb5b5a936a56945 [id=d3363ba8a35846b5bd1ae2b334e713cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.933866 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbed9dd56e94cad8d7387155b934c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.949393 2376 ts_tablet_manager.cc:1929] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:39.950846 2376 log.cc:1199] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/865fbd674db34f668fb5a2f5836a9b43
I20251026 02:56:39.952136 2376 ts_tablet_manager.cc:1950] T 865fbd674db34f668fb5a2f5836a9b43 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:39.953222 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 644) took 3944 ms. Trace:
I20251026 02:56:39.953866 2233 tablet_service.cc:1505] Processing CreateTablet for tablet f502f8edc2634e9288def155335ab1e3 (DEFAULT_TABLE table=default.Table_8466ce8a41f84c3797f015d455b86d1c [id=569b458344a740a880cf0c90077a70de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.954289 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f502f8edc2634e9288def155335ab1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.953447 2376 rpcz_store.cc:276] 1026 02:56:36.008671 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.008739 (+ 68us) service_pool.cc:225] Handling call
1026 02:56:39.836420 (+3827681us) ts_tablet_manager.cc:1151] Deleting tablet 865fbd674db34f668fb5a2f5836a9b43
1026 02:56:39.836434 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.846954 (+ 10520us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.948939 (+101985us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.953181 (+ 4242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3827378}
I20251026 02:56:39.959749 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 645) took 3899 ms. Trace:
I20251026 02:56:39.959946 2375 tablet_bootstrap.cc:492] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:39.964696 3639 tablet.cc:1722] T 8f3ae3a377af46449c3faca339f86053 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:39.963052 2376 rpcz_store.cc:276] 1026 02:56:36.060491 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.060578 (+ 87us) service_pool.cc:225] Handling call
1026 02:56:39.958992 (+3898414us) ts_tablet_manager.cc:1151] Deleting tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3
1026 02:56:39.959021 (+ 29us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.959709 (+ 688us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3898208}
I20251026 02:56:39.967602 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 646) took 3906 ms. Trace:
I20251026 02:56:39.967772 2376 rpcz_store.cc:276] 1026 02:56:36.060633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.060686 (+ 53us) service_pool.cc:225] Handling call
1026 02:56:39.966096 (+3905410us) ts_tablet_manager.cc:1151] Deleting tablet b8a1244c1c49471fb7711a2ca9b89a99
1026 02:56:39.966107 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.967581 (+ 1474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3905193}
I20251026 02:56:39.974485 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 656) took 3454 ms. Trace:
I20251026 02:56:39.974625 2376 rpcz_store.cc:276] 1026 02:56:36.520495 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.520579 (+ 84us) service_pool.cc:225] Handling call
1026 02:56:39.974214 (+3453635us) ts_tablet_manager.cc:1151] Deleting tablet 1c740ab6481e4481b95878c3a185129d
1026 02:56:39.974229 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.974438 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3411135}
I20251026 02:56:39.975733 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 657) took 3452 ms. Trace:
I20251026 02:56:39.975873 2376 rpcz_store.cc:276] 1026 02:56:36.522923 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.524366 (+ 1443us) service_pool.cc:225] Handling call
1026 02:56:39.975528 (+3451162us) ts_tablet_manager.cc:1151] Deleting tablet 64415f5588db4df0a94676b650fb3631
1026 02:56:39.975538 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.975684 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3412310}
I20251026 02:56:39.976531 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 658) took 3363 ms. Trace:
I20251026 02:56:39.976662 2376 rpcz_store.cc:276] 1026 02:56:36.613501 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.614344 (+ 843us) service_pool.cc:225] Handling call
1026 02:56:39.976356 (+3362012us) ts_tablet_manager.cc:1151] Deleting tablet c13d89fc829049428363ede24c350ee5
1026 02:56:39.976371 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.976493 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3361731}
I20251026 02:56:39.977283 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 659) took 3350 ms. Trace:
I20251026 02:56:39.977408 2376 rpcz_store.cc:276] 1026 02:56:36.627053 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.627120 (+ 67us) service_pool.cc:225] Handling call
1026 02:56:39.977093 (+3349973us) ts_tablet_manager.cc:1151] Deleting tablet f719c720cc474181a63786a3e248ab2c
1026 02:56:39.977102 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.977245 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3349709}
I20251026 02:56:39.977892 2376 tablet_replica.cc:333] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:39.978272 2376 raft_consensus.cc:2243] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:39.978622 2376 raft_consensus.cc:2272] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:39.980300 2376 ts_tablet_manager.cc:1916] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:39.988611 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 868dbbfce40c4dd6b36cc220966b1986 consensus state has no leader
I20251026 02:56:39.993319 1659 catalog_manager.cc:5649] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:39.994673 3239 raft_consensus.cc:493] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:39.994953 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet e9cd10092aac40068a1c7f85e514fd37 (table default.Table_fe09be2527a840b09e3e64c27965845d [id=de5272bcc29644208ddd95bdec21254e]). Expected version 1 got 0
I20251026 02:56:39.994884 3239 raft_consensus.cc:515] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:39.999893 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 234112c1e45948ec81f67e17639be32d (DEFAULT_TABLE table=default.Table_23d0e02f7246407f8337ddd38381fa9c [id=5b4866df0e9442f899850a0e47e90b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.000319 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234112c1e45948ec81f67e17639be32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.000809 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a3491b303f94fe2a2c838297f73d464" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:40.003851 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 63f1abc4477f4198ad2cc8481d69df73 (DEFAULT_TABLE table=default.Table_7ba3891cfe0b46329d460a7a3881d8d0 [id=3cd62b627c1d4b5d875e4bf0a9158aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.004066 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 63f1abc4477f4198ad2cc8481d69df73 (DEFAULT_TABLE table=default.Table_7ba3891cfe0b46329d460a7a3881d8d0 [id=3cd62b627c1d4b5d875e4bf0a9158aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.004328 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63f1abc4477f4198ad2cc8481d69df73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.004485 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63f1abc4477f4198ad2cc8481d69df73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.005573 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 6d788b5877bc49b99beb0ce511347de4 (DEFAULT_TABLE table=default.Table_e20051ad208b42e39e779968848a4c86 [id=532f5081abbe48b5a5e9773fe1b1046b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.005920 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d788b5877bc49b99beb0ce511347de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.005836 2227 tablet_service.cc:1505] Processing CreateTablet for tablet ecbed9dd56e94cad8d7387155b934c50 (DEFAULT_TABLE table=default.Table_07a34faac32549c99cb5b5a936a56945 [id=d3363ba8a35846b5bd1ae2b334e713cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.006181 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbed9dd56e94cad8d7387155b934c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.006410 3239 leader_election.cc:290] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.006930 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 663076e362c44837beb322aa965cde42 (DEFAULT_TABLE table=default.Table_888cc00019e343279dac7386f384975b [id=5e0db6c1b00d446d8366e36e146776df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.007301 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663076e362c44837beb322aa965cde42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.007640 1951 tablet_service.cc:1505] Processing CreateTablet for tablet dfa46c0ef7c84d03987f435408131469 (DEFAULT_TABLE table=default.Table_b7f76ac1487446019420d43442bcd4ae [id=717ffb32a1034080b508076eca0931f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.007956 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa46c0ef7c84d03987f435408131469. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.008734 1956 tablet_service.cc:1505] Processing CreateTablet for tablet e9bf547b01394be9b126de89c501193b (DEFAULT_TABLE table=default.Table_cbd8bd7117a34f5a836578a05d997586 [id=538895b1c1d14a519eaeec14e31d1210]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.009006 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9bf547b01394be9b126de89c501193b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.010020 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 865fbd674db34f668fb5a2f5836a9b43 (table default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa]) successfully deleted
W20251026 02:56:40.010303 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 already in progress: creating tablet
W20251026 02:56:40.010500 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b8a1244c1c49471fb7711a2ca9b89a99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8a1244c1c49471fb7711a2ca9b89a99 already in progress: creating tablet
W20251026 02:56:40.010686 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 1c740ab6481e4481b95878c3a185129d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c740ab6481e4481b95878c3a185129d already in progress: creating tablet
W20251026 02:56:40.010856 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 64415f5588db4df0a94676b650fb3631 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64415f5588db4df0a94676b650fb3631 already in progress: creating tablet
W20251026 02:56:40.011063 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet c13d89fc829049428363ede24c350ee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c13d89fc829049428363ede24c350ee5 already in progress: creating tablet
W20251026 02:56:40.011257 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet f719c720cc474181a63786a3e248ab2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f719c720cc474181a63786a3e248ab2c already in progress: creating tablet
I20251026 02:56:40.012856 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 6d788b5877bc49b99beb0ce511347de4 (DEFAULT_TABLE table=default.Table_e20051ad208b42e39e779968848a4c86 [id=532f5081abbe48b5a5e9773fe1b1046b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.966401 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbed9dd56e94cad8d7387155b934c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.974268 3651 tablet.cc:1722] T 8f3ae3a377af46449c3faca339f86053 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:40.014449 2375 ts_tablet_manager.cc:1403] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.132s user 0.005s sys 0.000s
I20251026 02:56:40.014734 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 8a076043f05c42d9afd9699fb4bec5ba (DEFAULT_TABLE table=default.Table_bd39192739b24c9ba278c4706acb20bb [id=96bea58b27fd4044a844df1a7f9b899b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.015041 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a076043f05c42d9afd9699fb4bec5ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:40.015255 2194 leader_election.cc:343] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.015244 2375 raft_consensus.cc:359] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.015403 2375 raft_consensus.cc:385] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.015465 2375 raft_consensus.cc:740] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.015686 2375 consensus_queue.cc:260] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.016322 2375 ts_tablet_manager.cc:1434] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:40.016494 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234112c1e45948ec81f67e17639be32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.018339 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 8ca8189991574d7c9d432dd737fee13e (DEFAULT_TABLE table=default.Table_c6376c7fc54d43c68859ff371a420894 [id=b492719c116d4b08a856f6f958da08a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.018671 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca8189991574d7c9d432dd737fee13e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:40.022850 2053 consensus_peers.cc:597] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:40.027211 2375 tablet_bootstrap.cc:492] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:40.028858 2375 tablet_bootstrap.cc:654] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.029570 1982 raft_consensus.cc:3060] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.034590 2228 tablet_service.cc:1505] Processing CreateTablet for tablet e9bf547b01394be9b126de89c501193b (DEFAULT_TABLE table=default.Table_cbd8bd7117a34f5a836578a05d997586 [id=538895b1c1d14a519eaeec14e31d1210]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.034970 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9bf547b01394be9b126de89c501193b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.036898 1978 raft_consensus.cc:3060] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.037457 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d788b5877bc49b99beb0ce511347de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.038113 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 8a076043f05c42d9afd9699fb4bec5ba (DEFAULT_TABLE table=default.Table_bd39192739b24c9ba278c4706acb20bb [id=96bea58b27fd4044a844df1a7f9b899b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.038468 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a076043f05c42d9afd9699fb4bec5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:39.991588 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 577) took 6452 ms. Trace:
I20251026 02:56:39.996001 2097 tablet_service.cc:1505] Processing CreateTablet for tablet e9bf547b01394be9b126de89c501193b (DEFAULT_TABLE table=default.Table_cbd8bd7117a34f5a836578a05d997586 [id=538895b1c1d14a519eaeec14e31d1210]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.996496 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 8a076043f05c42d9afd9699fb4bec5ba (DEFAULT_TABLE table=default.Table_bd39192739b24c9ba278c4706acb20bb [id=96bea58b27fd4044a844df1a7f9b899b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.996941 2099 tablet_service.cc:1505] Processing CreateTablet for tablet dfa46c0ef7c84d03987f435408131469 (DEFAULT_TABLE table=default.Table_b7f76ac1487446019420d43442bcd4ae [id=717ffb32a1034080b508076eca0931f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:39.997385 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 663076e362c44837beb322aa965cde42 (DEFAULT_TABLE table=default.Table_888cc00019e343279dac7386f384975b [id=5e0db6c1b00d446d8366e36e146776df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.013023 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a3491b303f94fe2a2c838297f73d464" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:40.039680 2118 raft_consensus.cc:2468] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:40.040198 2195 leader_election.cc:304] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:40.040649 3622 raft_consensus.cc:2804] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:40.040727 3622 raft_consensus.cc:493] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:40.040797 3622 raft_consensus.cc:3060] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.042712 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9bf547b01394be9b126de89c501193b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.047226 3622 raft_consensus.cc:515] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.047349 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a076043f05c42d9afd9699fb4bec5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.047828 3622 leader_election.cc:290] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.053494 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa46c0ef7c84d03987f435408131469. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.060706 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a3491b303f94fe2a2c838297f73d464" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:40.060914 2118 raft_consensus.cc:3060] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.063299 2377 ts_tablet_manager.cc:1929] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.063402 2377 log.cc:1199] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/f6d6f024d8b943d5acf69b64bc7c7214
I20251026 02:56:40.063858 2377 ts_tablet_manager.cc:1950] T f6d6f024d8b943d5acf69b64bc7c7214 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:40.063954 2118 raft_consensus.cc:2468] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:40.067631 2195 leader_election.cc:304] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:40.067904 3239 raft_consensus.cc:2804] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:40.068022 3239 raft_consensus.cc:697] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:40.068272 3239 consensus_queue.cc:237] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.071883 2381 tablet_bootstrap.cc:492] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:40.072621 2381 ts_tablet_manager.cc:1403] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.364s user 0.012s sys 0.017s
I20251026 02:56:40.073871 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663076e362c44837beb322aa965cde42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.076532 2381 raft_consensus.cc:359] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.077287 2381 raft_consensus.cc:385] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.078104 2381 raft_consensus.cc:740] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.079110 2381 consensus_queue.cc:260] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.080353 2237 tablet_service.cc:1505] Processing CreateTablet for tablet dfa46c0ef7c84d03987f435408131469 (DEFAULT_TABLE table=default.Table_b7f76ac1487446019420d43442bcd4ae [id=717ffb32a1034080b508076eca0931f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.077338 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 663076e362c44837beb322aa965cde42 (DEFAULT_TABLE table=default.Table_888cc00019e343279dac7386f384975b [id=5e0db6c1b00d446d8366e36e146776df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.083732 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a3491b303f94fe2a2c838297f73d464" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:40.084357 2194 leader_election.cc:343] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.039391 2372 rpcz_store.cc:276] 1026 02:56:33.538979 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.539025 (+ 46us) service_pool.cc:225] Handling call
1026 02:56:39.296615 (+5757590us) ts_tablet_manager.cc:1151] Deleting tablet 47037db3602246a2ab6224557286162f
1026 02:56:39.304070 (+ 7455us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:39.839086 (+535016us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.907241 (+ 68155us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:39.991563 (+ 84322us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5757392}
I20251026 02:56:40.089952 2372 tablet_replica.cc:333] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:40.090068 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 47037db3602246a2ab6224557286162f (table default.Table_09ca38422af543f89eab26b4117536dc [id=12a8deac66914c6da970cd7f975b5b59]) successfully deleted
I20251026 02:56:40.090216 2372 raft_consensus.cc:2243] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:40.090466 2372 pending_rounds.cc:70] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48: Trying to abort 1 pending ops.
I20251026 02:56:40.090540 2372 pending_rounds.cc:77] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888548657512448 op_type: NO_OP noop_request { }
I20251026 02:56:40.092278 2372 raft_consensus.cc:2889] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:40.092429 2372 raft_consensus.cc:2272] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.093655 2372 ts_tablet_manager.cc:1916] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.096760 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 8ca8189991574d7c9d432dd737fee13e (DEFAULT_TABLE table=default.Table_c6376c7fc54d43c68859ff371a420894 [id=b492719c116d4b08a856f6f958da08a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.097155 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca8189991574d7c9d432dd737fee13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.097630 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 234112c1e45948ec81f67e17639be32d (DEFAULT_TABLE table=default.Table_23d0e02f7246407f8337ddd38381fa9c [id=5b4866df0e9442f899850a0e47e90b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.097581 2381 ts_tablet_manager.cc:1434] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20251026 02:56:40.098752 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 8ca8189991574d7c9d432dd737fee13e (DEFAULT_TABLE table=default.Table_c6376c7fc54d43c68859ff371a420894 [id=b492719c116d4b08a856f6f958da08a0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:40.100834 2195 consensus_peers.cc:597] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be1c0948c0ae4563b7cb644521914cb9. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:40.105365 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 7c148c74fbcd4d30b9ba0ea059284427 (DEFAULT_TABLE table=default.Table_dc1b77e5238847adbd80a2df2274c003 [id=81ecbd0df65d45e4b8f1af69ca4fd488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.105801 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c148c74fbcd4d30b9ba0ea059284427. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:40.112048 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3aea01a248b345cf81d4d9862744eb78 consensus state has no leader
W20251026 02:56:40.114703 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 886fe7ffa394440eb893887a9feaff2b has no consensus state
I20251026 02:56:40.114982 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet d9391e7654c442a5bb312f0daaf1d99b from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:40.117571 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 046a34f9408b406ba08471421f422064 (table default.Table_760b799491d7426a86aa1dc3c76fab31 [id=bc8a2d95ed134fae9f3e9deeac4f130a]). Expected version 3 got 0
I20251026 02:56:40.117784 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 8f3ae3a377af46449c3faca339f86053 (table default.Table_3af0e3e691254fbb8f124697619830bf [id=31f977e55ba041f59bcc721d469746fd]). Expected version 3 got 0
I20251026 02:56:40.119095 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet e701d4023fee45d18c48997aab3227c2 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:40.119701 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_675403100cd74e78a423ffac082f4e8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.120276 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_675403100cd74e78a423ffac082f4e8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.121454 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0e6ed600363d4be391d9f26eaa01e256" } new_table_name: "default.Table_7262f683830a4f35ae728338fc3344b8" modify_external_catalogs: true
I20251026 02:56:40.122749 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_61f1d10f16b1467aaab14fe4fbaef1bc" } new_table_name: "default.Table_1821574ebc7f452b9cc0d8231f78cbdc" modify_external_catalogs: true
I20251026 02:56:40.130245 1980 raft_consensus.cc:3060] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.136600 1974 raft_consensus.cc:3060] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.139488 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 561) took 7082 ms. Trace:
I20251026 02:56:40.139595 2377 rpcz_store.cc:276] 1026 02:56:33.057248 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.057291 (+ 43us) service_pool.cc:225] Handling call
1026 02:56:39.856878 (+6799587us) ts_tablet_manager.cc:1151] Deleting tablet f6d6f024d8b943d5acf69b64bc7c7214
1026 02:56:39.856885 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.025830 (+168945us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.063212 (+ 37382us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.139466 (+ 76254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6799458}
I20251026 02:56:40.139962 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 562) took 7082 ms. Trace:
I20251026 02:56:40.140033 2377 rpcz_store.cc:276] 1026 02:56:33.057655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.057694 (+ 39us) service_pool.cc:225] Handling call
1026 02:56:40.139869 (+7082175us) ts_tablet_manager.cc:1151] Deleting tablet 122de8616b8f4fcf88971d5738c47e82
1026 02:56:40.139879 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.139948 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7082041}
I20251026 02:56:40.140307 2377 tablet_replica.cc:333] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:40.140547 2377 raft_consensus.cc:2243] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:40.140889 2375 tablet_bootstrap.cc:492] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:40.140993 2377 raft_consensus.cc:2272] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.141037 2375 ts_tablet_manager.cc:1403] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.114s user 0.003s sys 0.000s
I20251026 02:56:40.141527 2377 ts_tablet_manager.cc:1916] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.141775 2375 raft_consensus.cc:359] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.141948 2375 raft_consensus.cc:385] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.141997 2375 raft_consensus.cc:740] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.142228 2375 consensus_queue.cc:260] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.142735 2375 ts_tablet_manager.cc:1434] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:40.143009 2375 tablet_bootstrap.cc:492] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:40.143225 2372 ts_tablet_manager.cc:1929] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.143311 2372 log.cc:1199] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/dbf2a1ce78154415b23ab6d0efbc1f52
I20251026 02:56:40.143625 2372 ts_tablet_manager.cc:1950] T dbf2a1ce78154415b23ab6d0efbc1f52 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:40.143884 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 576) took 6605 ms. Trace:
I20251026 02:56:40.143942 2372 rpcz_store.cc:276] 1026 02:56:33.538762 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.540262 (+ 1500us) service_pool.cc:225] Handling call
1026 02:56:40.089897 (+6549635us) ts_tablet_manager.cc:1151] Deleting tablet dbf2a1ce78154415b23ab6d0efbc1f52
1026 02:56:40.089908 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.104513 (+ 14605us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.143117 (+ 38604us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.143870 (+ 753us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6549467}
I20251026 02:56:40.144848 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa46c0ef7c84d03987f435408131469. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.146952 2372 tablet_replica.cc:333] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:40.147152 2372 raft_consensus.cc:2243] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:40.147317 2372 raft_consensus.cc:2272] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.148126 2372 ts_tablet_manager.cc:1916] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.148380 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet dbf2a1ce78154415b23ab6d0efbc1f52 (table default.Table_22ae79375e8e49d4b6d146d087ffa64b [id=4c170eaba0fd422f9fb0ce5e63e5c746]) successfully deleted
I20251026 02:56:40.149719 2375 tablet_bootstrap.cc:654] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.151218 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663076e362c44837beb322aa965cde42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.158468 2372 ts_tablet_manager.cc:1929] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.160827 2372 log.cc:1199] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/17d2cc6b16d34331bb80c576c0cd2ce0
I20251026 02:56:40.161269 2372 ts_tablet_manager.cc:1950] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:40.162111 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 578) took 6614 ms. Trace:
I20251026 02:56:40.162238 2372 rpcz_store.cc:276] 1026 02:56:33.547522 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.547587 (+ 65us) service_pool.cc:225] Handling call
1026 02:56:40.144189 (+6596602us) ts_tablet_manager.cc:1151] Deleting tablet 17d2cc6b16d34331bb80c576c0cd2ce0
1026 02:56:40.144758 (+ 569us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.151974 (+ 7216us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.158347 (+ 6373us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.162094 (+ 3747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6596388}
I20251026 02:56:40.162791 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 17d2cc6b16d34331bb80c576c0cd2ce0 (table default.Table_9f3815b1bf464a83a574e3075d065956 [id=095ad208529348d4aea2d4e2840e07c7]) successfully deleted
I20251026 02:56:40.163156 2372 tablet_replica.cc:333] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:40.163388 2372 raft_consensus.cc:2243] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:40.163597 2372 raft_consensus.cc:2272] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.167395 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234112c1e45948ec81f67e17639be32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.172151 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 7c148c74fbcd4d30b9ba0ea059284427 (DEFAULT_TABLE table=default.Table_dc1b77e5238847adbd80a2df2274c003 [id=81ecbd0df65d45e4b8f1af69ca4fd488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.172605 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c148c74fbcd4d30b9ba0ea059284427. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.176653 2372 ts_tablet_manager.cc:1916] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.177161 2378 tablet_bootstrap.cc:492] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:40.177404 2378 ts_tablet_manager.cc:1403] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.312s user 0.005s sys 0.002s
I20251026 02:56:40.184747 2377 ts_tablet_manager.cc:1929] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.185119 2377 log.cc:1199] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/863ce9e1adb444b7b1399b968ea79bb4
I20251026 02:56:40.185715 2377 ts_tablet_manager.cc:1950] T 863ce9e1adb444b7b1399b968ea79bb4 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:40.187732 2378 raft_consensus.cc:359] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.187937 2378 raft_consensus.cc:385] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.187994 2378 raft_consensus.cc:740] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.191685 1700 master_runner.cc:186] Error getting master registration for 127.1.150.60:33729: OK, instance_id { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" instance_seqno: 1761447396760524 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251026 02:56:40.191958 1700 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 02:56:40.192365 2378 consensus_queue.cc:260] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.192566 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet 45e79be9b98945c49fb712019b21b91b from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:40.193245 2378 ts_tablet_manager.cc:1434] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251026 02:56:40.195132 2376 ts_tablet_manager.cc:1929] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.195308 2376 log.cc:1199] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/03da415f3a944a9a9679b3a45295ce35
I20251026 02:56:40.195971 2376 ts_tablet_manager.cc:1950] T 03da415f3a944a9a9679b3a45295ce35 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:40.198719 2381 tablet_bootstrap.cc:492] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:40.200130 2381 tablet_bootstrap.cc:654] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.200762 2378 tablet_bootstrap.cc:492] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:40.201028 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca8189991574d7c9d432dd737fee13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.203361 2378 tablet_bootstrap.cc:654] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.204380 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet f6d6f024d8b943d5acf69b64bc7c7214 (table default.Table_30b96d2d4369458390803c6be18efe0e [id=9fd6e480fc3543149959773e691a3ddd]) successfully deleted
I20251026 02:56:40.204762 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 075f6ca8787943a09443863c544a1284 (DEFAULT_TABLE table=default.Table_0e0f079c121342b9bb342f26e3c24518 [id=e6764201b957439789459cc8882dfe71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.205117 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075f6ca8787943a09443863c544a1284. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.205034 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 075f6ca8787943a09443863c544a1284 (DEFAULT_TABLE table=default.Table_0e0f079c121342b9bb342f26e3c24518 [id=e6764201b957439789459cc8882dfe71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.205443 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075f6ca8787943a09443863c544a1284. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.209764 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 7c148c74fbcd4d30b9ba0ea059284427 (DEFAULT_TABLE table=default.Table_dc1b77e5238847adbd80a2df2274c003 [id=81ecbd0df65d45e4b8f1af69ca4fd488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.210121 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c148c74fbcd4d30b9ba0ea059284427. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.210110 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 663) took 3484 ms. Trace:
I20251026 02:56:40.210234 2376 rpcz_store.cc:276] 1026 02:56:36.725855 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.726037 (+ 182us) service_pool.cc:225] Handling call
1026 02:56:39.977817 (+3251780us) ts_tablet_manager.cc:1151] Deleting tablet 03da415f3a944a9a9679b3a45295ce35
1026 02:56:39.977826 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.085714 (+107888us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.194983 (+109269us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.210083 (+ 15100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12160,"tablet-delete.queue_time_us":3222231}
I20251026 02:56:40.212587 3372 raft_consensus.cc:493] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.212743 3372 raft_consensus.cc:515] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.213471 3372 leader_election.cc:290] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:40.214692 1916 leader_election.cc:343] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 47037db3602246a2ab6224557286162f
I20251026 02:56:40.215451 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:40.215670 2257 raft_consensus.cc:2393] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:40.216069 1915 leader_election.cc:304] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:40.216663 3372 raft_consensus.cc:3060] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.219973 3372 raft_consensus.cc:2749] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:40.215785 2376 tablet_replica.cc:333] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:40.220844 2376 raft_consensus.cc:2243] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:40.222407 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 563) took 7164 ms. Trace:
I20251026 02:56:40.222599 2377 rpcz_store.cc:276] 1026 02:56:33.058022 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.058073 (+ 51us) service_pool.cc:225] Handling call
1026 02:56:40.140241 (+7082168us) ts_tablet_manager.cc:1151] Deleting tablet 863ce9e1adb444b7b1399b968ea79bb4
1026 02:56:40.140266 (+ 25us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.181206 (+ 40940us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.184633 (+ 3427us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.222382 (+ 37749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7082032}
I20251026 02:56:40.222813 2376 raft_consensus.cc:2272] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.223016 2377 tablet_replica.cc:333] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:40.224869 2378 tablet_bootstrap.cc:492] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:40.225065 2378 ts_tablet_manager.cc:1403] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.003s
I20251026 02:56:40.225536 3239 raft_consensus.cc:493] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.226035 2378 raft_consensus.cc:359] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.226207 2378 raft_consensus.cc:385] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.226261 2378 raft_consensus.cc:740] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.226233 3239 raft_consensus.cc:515] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.226485 2378 consensus_queue.cc:260] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.226856 3239 leader_election.cc:290] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.227497 2378 ts_tablet_manager.cc:1434] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 02:56:40.227860 2378 tablet_bootstrap.cc:492] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:40.224186 2376 ts_tablet_manager.cc:1916] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.231468 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 863ce9e1adb444b7b1399b968ea79bb4 (table default.Table_03bc36a9fd184c28abd1caf6e5ec4f7a [id=ee13c495b56c48649ceacc8e1122efb4]) successfully deleted
I20251026 02:56:40.237779 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 7096b78402074049b9c8ed1b9fe21f0e (DEFAULT_TABLE table=default.Table_56c374c6abad4892834544f5df51a5db [id=33917bc244244b29973e28fe82515bca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.238240 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7096b78402074049b9c8ed1b9fe21f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.238905 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7801f073d5d94076af9299492f86fad2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:40.239082 2118 raft_consensus.cc:2468] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:40.239193 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 6d788b5877bc49b99beb0ce511347de4 (DEFAULT_TABLE table=default.Table_e20051ad208b42e39e779968848a4c86 [id=532f5081abbe48b5a5e9773fe1b1046b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.239502 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d788b5877bc49b99beb0ce511347de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.239511 2195 leader_election.cc:304] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:40.239964 3521 raft_consensus.cc:2804] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:40.240036 3521 raft_consensus.cc:493] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:40.240090 3521 raft_consensus.cc:3060] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.241075 3622 raft_consensus.cc:493] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.241219 3622 raft_consensus.cc:515] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.241751 3622 leader_election.cc:290] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.242578 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b7d85a2941545ceaa16b22e8328b4ff" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:40.242700 3521 raft_consensus.cc:515] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.243199 3521 leader_election.cc:290] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.243443 3521 raft_consensus.cc:493] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.243603 3521 raft_consensus.cc:515] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.244153 3521 leader_election.cc:290] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.244267 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7801f073d5d94076af9299492f86fad2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:40.244477 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 03da415f3a944a9a9679b3a45295ce35 (table default.Table_828c16a96a47489f835a891f34d707b1 [id=432234d273ca4c9ea53c7a078c274637]) successfully deleted
I20251026 02:56:40.244308 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 706036515b454880b3413a994c5d1138 (DEFAULT_TABLE table=default.Table_12c63af0fb4a44b19e48f3ecb496a2d2 [id=699d1decca35472086f70ffa969fe399]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.244958 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 706036515b454880b3413a994c5d1138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.244937 2377 raft_consensus.cc:2243] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:40.246291 2377 raft_consensus.cc:2272] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.246829 2377 ts_tablet_manager.cc:1916] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.251521 2378 tablet_bootstrap.cc:654] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.252727 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7801f073d5d94076af9299492f86fad2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:40.253293 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 7096b78402074049b9c8ed1b9fe21f0e (DEFAULT_TABLE table=default.Table_56c374c6abad4892834544f5df51a5db [id=33917bc244244b29973e28fe82515bca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.253676 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7096b78402074049b9c8ed1b9fe21f0e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:40.254106 2194 leader_election.cc:343] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.254781 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85f9fcae83a04c6e907c7e9c8cd95dbe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:40.255005 2194 leader_election.cc:343] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.255043 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 706036515b454880b3413a994c5d1138 (DEFAULT_TABLE table=default.Table_12c63af0fb4a44b19e48f3ecb496a2d2 [id=699d1decca35472086f70ffa969fe399]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.255347 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 706036515b454880b3413a994c5d1138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.255488 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 7096b78402074049b9c8ed1b9fe21f0e (DEFAULT_TABLE table=default.Table_56c374c6abad4892834544f5df51a5db [id=33917bc244244b29973e28fe82515bca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.255795 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7096b78402074049b9c8ed1b9fe21f0e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:40.256382 2194 leader_election.cc:343] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.259040 2376 ts_tablet_manager.cc:1929] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.259171 2376 log.cc:1199] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/ced7ad6a601d432cbc13b3199a0506e4
I20251026 02:56:40.259508 2376 ts_tablet_manager.cc:1950] T ced7ad6a601d432cbc13b3199a0506e4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:40.260358 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 664) took 3534 ms. Trace:
I20251026 02:56:40.260469 2376 rpcz_store.cc:276] 1026 02:56:36.725978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.726218 (+ 240us) service_pool.cc:225] Handling call
1026 02:56:40.215735 (+3489517us) ts_tablet_manager.cc:1151] Deleting tablet ced7ad6a601d432cbc13b3199a0506e4
1026 02:56:40.215748 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.255958 (+ 40210us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.258930 (+ 2972us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.260309 (+ 1379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3460064}
I20251026 02:56:40.260819 2376 tablet_replica.cc:333] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:40.261026 2376 raft_consensus.cc:2243] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:40.261269 2376 raft_consensus.cc:2272] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.261899 2376 ts_tablet_manager.cc:1916] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.262615 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b7d85a2941545ceaa16b22e8328b4ff" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:40.263001 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet ced7ad6a601d432cbc13b3199a0506e4 (table default.Table_241789d6111d497c89efdb4507dff32e [id=4e7fddec9b124b32b7786ea15111878b]) successfully deleted
W20251026 02:56:40.263669 2194 leader_election.cc:343] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.242744 2118 raft_consensus.cc:2468] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:40.243589 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7801f073d5d94076af9299492f86fad2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:40.275609 2377 ts_tablet_manager.cc:1929] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.275660 2111 raft_consensus.cc:3060] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.275738 2377 log.cc:1199] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/ced7ad6a601d432cbc13b3199a0506e4
W20251026 02:56:40.262315 2053 consensus_peers.cc:597] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7062c5ff91e14369ab69a2d199585c55. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:40.244755 2107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85f9fcae83a04c6e907c7e9c8cd95dbe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:40.276193 2107 raft_consensus.cc:2468] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:40.268597 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 706036515b454880b3413a994c5d1138 (DEFAULT_TABLE table=default.Table_12c63af0fb4a44b19e48f3ecb496a2d2 [id=699d1decca35472086f70ffa969fe399]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.245622 2372 ts_tablet_manager.cc:1929] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.276718 2372 log.cc:1199] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/ce922040298d4952af13652eeda7a234
I20251026 02:56:40.277037 2372 ts_tablet_manager.cc:1950] T ce922040298d4952af13652eeda7a234 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:40.277698 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 706036515b454880b3413a994c5d1138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.278091 2195 leader_election.cc:304] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:40.278532 2195 leader_election.cc:304] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:40.278947 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2f1ecb7efe6244588067d3c2ba9be9ad" } modify_external_catalogs: true
I20251026 02:56:40.279042 3239 raft_consensus.cc:2804] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:40.279116 3239 raft_consensus.cc:493] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:40.279192 3239 raft_consensus.cc:3060] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.279209 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2f1ecb7efe6244588067d3c2ba9be9ad" } modify_external_catalogs: true
I20251026 02:56:40.280314 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 579) took 6732 ms. Trace:
I20251026 02:56:40.280412 2372 rpcz_store.cc:276] 1026 02:56:33.547951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.548000 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:40.163094 (+6615094us) ts_tablet_manager.cc:1151] Deleting tablet ce922040298d4952af13652eeda7a234
1026 02:56:40.163103 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.240486 (+ 77383us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.245447 (+ 4961us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.280295 (+ 34848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6614899}
I20251026 02:56:40.280652 3521 raft_consensus.cc:2804] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:40.280725 2372 tablet_replica.cc:333] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:40.280725 3521 raft_consensus.cc:493] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:40.280779 3521 raft_consensus.cc:3060] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.280906 2372 raft_consensus.cc:2243] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:40.281224 2372 raft_consensus.cc:2272] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.282049 3239 raft_consensus.cc:515] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.282176 2372 ts_tablet_manager.cc:1916] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.283116 3239 leader_election.cc:290] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.283504 3521 raft_consensus.cc:515] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.283737 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 075f6ca8787943a09443863c544a1284 (DEFAULT_TABLE table=default.Table_0e0f079c121342b9bb342f26e3c24518 [id=e6764201b957439789459cc8882dfe71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.284098 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075f6ca8787943a09443863c544a1284. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.284540 3521 leader_election.cc:290] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.284731 2107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b7d85a2941545ceaa16b22e8328b4ff" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:40.284898 2107 raft_consensus.cc:3060] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.284991 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85f9fcae83a04c6e907c7e9c8cd95dbe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:40.285148 2118 raft_consensus.cc:3060] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.287983 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet d9391e7654c442a5bb312f0daaf1d99b
I20251026 02:56:40.288555 1662 catalog_manager.cc:6719] ReplaceTablet: tablet d9391e7654c442a5bb312f0daaf1d99b deleted and replaced by tablet e45600a92f8e46b096badd700779856d
I20251026 02:56:40.289036 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet d9391e7654c442a5bb312f0daaf1d99b with delete_type TABLET_DATA_DELETED (replaced by tablet e45600a92f8e46b096badd700779856d) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.293319 2381 tablet_bootstrap.cc:492] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:40.293588 2381 ts_tablet_manager.cc:1403] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.001s
I20251026 02:56:40.294245 2111 raft_consensus.cc:2468] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:40.294514 2381 raft_consensus.cc:359] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.294761 2381 raft_consensus.cc:385] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.294903 2381 raft_consensus.cc:740] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.295135 2381 consensus_queue.cc:260] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.296097 2381 ts_tablet_manager.cc:1434] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:40.296430 2381 tablet_bootstrap.cc:492] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:40.296548 2107 raft_consensus.cc:2468] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:40.298190 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 046a34f9408b406ba08471421f422064 (table default.Table_760b799491d7426a86aa1dc3c76fab31 [id=bc8a2d95ed134fae9f3e9deeac4f130a]). Expected version 3 got 0
I20251026 02:56:40.298655 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet dba102838945453cbf44e1cfe505a52c (table default.Table_8ffce6cf6b2d44aa8a9459e834de1d80 [id=92310ee99b5c428c9773b0f7a5462a4d]). Expected version 1 got 0
I20251026 02:56:40.298952 1660 catalog_manager.cc:5649] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:40.299450 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet c4b0933f71cb4ac7a58986d16b613e64 (table default.Table_e6b19dda39ad45baae642c27b07db42d [id=cba6d4517d244d7580e525729e871745]). Expected version 4 got 0
I20251026 02:56:40.300308 2381 tablet_bootstrap.cc:654] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:40.302713 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4d64902618d46428ff345bc190b03bd consensus state has no leader
W20251026 02:56:40.302898 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 consensus state has no leader
I20251026 02:56:40.303073 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 9b9a5daa36b547a7888d38dc3329eaf2 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:40.303467 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet e701d4023fee45d18c48997aab3227c2
I20251026 02:56:40.303544 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_42d9349693d44bc5bff21ffa159fd32f" } modify_external_catalogs: true
I20251026 02:56:40.303647 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_10236dd7e7264f888b32afcb8c56c95f" } modify_external_catalogs: true
I20251026 02:56:40.303742 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_42d9349693d44bc5bff21ffa159fd32f" } modify_external_catalogs: true
I20251026 02:56:40.303848 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_10236dd7e7264f888b32afcb8c56c95f" } modify_external_catalogs: true
I20251026 02:56:40.304050 1661 catalog_manager.cc:6719] ReplaceTablet: tablet e701d4023fee45d18c48997aab3227c2 deleted and replaced by tablet ac82c14e7dcb448b845d998249e2b339
I20251026 02:56:40.304414 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4e452bf0486241589709e97602ea9942"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.305011 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e452bf0486241589709e97602ea9942 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.305274 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_bd39192739b24c9ba278c4706acb20bb" } modify_external_catalogs: true
I20251026 02:56:40.305459 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_bd39192739b24c9ba278c4706acb20bb" } modify_external_catalogs: true
I20251026 02:56:40.307767 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_56c374c6abad4892834544f5df51a5db" } modify_external_catalogs: true
I20251026 02:56:40.307942 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_56c374c6abad4892834544f5df51a5db" } modify_external_catalogs: true
I20251026 02:56:40.308028 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet e701d4023fee45d18c48997aab3227c2 with delete_type TABLET_DATA_DELETED (replaced by tablet ac82c14e7dcb448b845d998249e2b339) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.313526 2118 raft_consensus.cc:2468] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:40.314348 2195 leader_election.cc:304] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:40.314633 3521 raft_consensus.cc:2804] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:40.314764 2195 leader_election.cc:304] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:40.314939 3239 raft_consensus.cc:2804] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:40.314970 2195 leader_election.cc:304] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:40.315028 3239 raft_consensus.cc:697] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:40.315219 3622 raft_consensus.cc:2804] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:40.315317 3622 raft_consensus.cc:697] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:40.315268 3239 consensus_queue.cc:237] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.315511 3622 consensus_queue.cc:237] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.315845 3547 raft_consensus.cc:493] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.316038 3547 raft_consensus.cc:515] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.317436 3547 leader_election.cc:290] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:40.314849 3521 raft_consensus.cc:697] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:40.320043 3521 consensus_queue.cc:237] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.320858 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a012d868a78f4324a92db83c81931b9d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:40.321758 2053 leader_election.cc:343] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:40.323099 2195 consensus_peers.cc:597] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce922040298d4952af13652eeda7a234. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:40.327699 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a012d868a78f4324a92db83c81931b9d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:40.328085 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet e701d4023fee45d18c48997aab3227c2 with delete_type TABLET_DATA_DELETED (replaced by tablet ac82c14e7dcb448b845d998249e2b339) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.328938 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 44af74033ccf4d048475f12d164938f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.329171 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet d9391e7654c442a5bb312f0daaf1d99b with delete_type TABLET_DATA_DELETED (replaced by tablet e45600a92f8e46b096badd700779856d) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.330582 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet e701d4023fee45d18c48997aab3227c2 with delete_type TABLET_DATA_DELETED (replaced by tablet ac82c14e7dcb448b845d998249e2b339) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.328366 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85f9fcae83a04c6e907c7e9c8cd95dbe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:40.331964 2053 leader_election.cc:343] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.332080 2053 leader_election.cc:304] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:40.332864 3647 raft_consensus.cc:2749] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:40.332955 2194 leader_election.cc:343] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:40.333411 2194 consensus_peers.cc:597] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 865fbd674db34f668fb5a2f5836a9b43. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:40.330883 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b7d85a2941545ceaa16b22e8328b4ff" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:40.334970 2194 leader_election.cc:343] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.342803 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet d9391e7654c442a5bb312f0daaf1d99b with delete_type TABLET_DATA_DELETED (replaced by tablet e45600a92f8e46b096badd700779856d) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.276181 2377 ts_tablet_manager.cc:1950] T ced7ad6a601d432cbc13b3199a0506e4 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:40.342507 2376 ts_tablet_manager.cc:1929] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:40.346453 2376 log.cc:1199] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/122de8616b8f4fcf88971d5738c47e82
I20251026 02:56:40.347529 2376 ts_tablet_manager.cc:1950] T 122de8616b8f4fcf88971d5738c47e82 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:40.348174 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 558) took 7292 ms. Trace:
I20251026 02:56:40.349815 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 662) took 3624 ms. Trace:
I20251026 02:56:40.350270 2377 rpcz_store.cc:276] 1026 02:56:33.055982 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.061673 (+ 5691us) service_pool.cc:225] Handling call
1026 02:56:40.222969 (+7161296us) ts_tablet_manager.cc:1151] Deleting tablet ced7ad6a601d432cbc13b3199a0506e4
1026 02:56:40.222979 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.271817 (+ 48838us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.275487 (+ 3670us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.348150 (+ 72663us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7161049}
I20251026 02:56:40.351154 2377 tablet_replica.cc:333] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:40.351368 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet ced7ad6a601d432cbc13b3199a0506e4 (table default.Table_241789d6111d497c89efdb4507dff32e [id=4e7fddec9b124b32b7786ea15111878b]) successfully deleted
I20251026 02:56:40.351725 2377 raft_consensus.cc:2243] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:40.352646 2376 rpcz_store.cc:276] 1026 02:56:36.725772 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.726528 (+ 756us) service_pool.cc:225] Handling call
1026 02:56:40.260738 (+3534210us) ts_tablet_manager.cc:1151] Deleting tablet 122de8616b8f4fcf88971d5738c47e82
1026 02:56:40.260751 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.332392 (+ 71641us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.342359 (+ 9967us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.349797 (+ 7438us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3499512}
I20251026 02:56:40.353255 2377 raft_consensus.cc:2272] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.353915 2375 tablet_bootstrap.cc:492] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:40.354202 2375 ts_tablet_manager.cc:1403] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.211s user 0.005s sys 0.017s
I20251026 02:56:40.357641 2375 raft_consensus.cc:359] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.358013 2375 raft_consensus.cc:385] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.358193 2375 raft_consensus.cc:740] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.358625 2375 consensus_queue.cc:260] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.359663 2375 ts_tablet_manager.cc:1434] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251026 02:56:40.360627 2372 ts_tablet_manager.cc:1929] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.364853 2377 ts_tablet_manager.cc:1916] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.365545 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 661) took 3644 ms. Trace:
I20251026 02:56:40.365742 2376 rpcz_store.cc:276] 1026 02:56:36.721292 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.739013 (+ 17721us) service_pool.cc:225] Handling call
1026 02:56:40.365410 (+3626397us) ts_tablet_manager.cc:1151] Deleting tablet 244c03a9293242a48077bdaf90a47865
1026 02:56:40.365420 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.365525 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3584124}
W20251026 02:56:40.366531 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 244c03a9293242a48077bdaf90a47865 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 244c03a9293242a48077bdaf90a47865 already in progress: creating tablet
W20251026 02:56:40.358412 2195 consensus_peers.cc:597] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17d2cc6b16d34331bb80c576c0cd2ce0. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:40.357075 2381 tablet_bootstrap.cc:492] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:40.367280 2381 ts_tablet_manager.cc:1403] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.001s
I20251026 02:56:40.368422 2381 raft_consensus.cc:359] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.368624 2381 raft_consensus.cc:385] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.368700 2381 raft_consensus.cc:740] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.368942 2381 consensus_queue.cc:260] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.369560 2381 ts_tablet_manager.cc:1434] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:40.369853 2381 tablet_bootstrap.cc:492] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:40.371968 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61
I20251026 02:56:40.372279 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet dba102838945453cbf44e1cfe505a52c (table default.Table_8ffce6cf6b2d44aa8a9459e834de1d80 [id=92310ee99b5c428c9773b0f7a5462a4d]). Expected version 1 got 0
I20251026 02:56:40.372610 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (table default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5]). Expected version 1 got 0
I20251026 02:56:40.372994 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.373091 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 046a34f9408b406ba08471421f422064 (table default.Table_760b799491d7426a86aa1dc3c76fab31 [id=bc8a2d95ed134fae9f3e9deeac4f130a]). Expected version 3 got 0
I20251026 02:56:40.373318 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet ff242ef9541a42d793c4484190956da6 (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]). Expected version 1 got 0
I20251026 02:56:40.373580 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (table default.Table_1fa437ef7f89450bb6f672a1fb491baa [id=44c37945ff9f48e8b2c07648394ac177]). Expected version 1 got 0
I20251026 02:56:40.373960 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.374418 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 45e79be9b98945c49fb712019b21b91b
I20251026 02:56:40.374974 1657 catalog_manager.cc:6719] ReplaceTablet: tablet 45e79be9b98945c49fb712019b21b91b deleted and replaced by tablet 1104345f80d6418fa907a8a26924c281
I20251026 02:56:40.375044 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 45e79be9b98945c49fb712019b21b91b with delete_type TABLET_DATA_DELETED (replaced by tablet 1104345f80d6418fa907a8a26924c281) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.375610 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 4e645650709f47eea4243ab64ec31312
I20251026 02:56:40.375865 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 45e79be9b98945c49fb712019b21b91b with delete_type TABLET_DATA_DELETED (replaced by tablet 1104345f80d6418fa907a8a26924c281) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.376535 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e645650709f47eea4243ab64ec31312 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.377421 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e645650709f47eea4243ab64ec31312 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.380709 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 660) took 3659 ms. Trace:
I20251026 02:56:40.381799 2381 tablet_bootstrap.cc:654] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.381722 2376 rpcz_store.cc:276] 1026 02:56:36.721173 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.744289 (+ 23116us) service_pool.cc:225] Handling call
1026 02:56:40.366921 (+3622632us) ts_tablet_manager.cc:1151] Deleting tablet 58d60bf2a9e546e6a9474a5af6d5273a
1026 02:56:40.366930 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.380688 (+ 13758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3585509}
W20251026 02:56:40.382223 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 58d60bf2a9e546e6a9474a5af6d5273a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58d60bf2a9e546e6a9474a5af6d5273a already in progress: creating tablet
I20251026 02:56:40.382968 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 665) took 3367 ms. Trace:
I20251026 02:56:40.383414 2376 rpcz_store.cc:276] 1026 02:56:37.015870 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.016291 (+ 421us) service_pool.cc:225] Handling call
1026 02:56:40.382830 (+3366539us) ts_tablet_manager.cc:1151] Deleting tablet ded06ba1f80c4433ae78687e5990edce
1026 02:56:40.382840 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.382952 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3366288}
W20251026 02:56:40.384109 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ded06ba1f80c4433ae78687e5990edce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ded06ba1f80c4433ae78687e5990edce already in progress: creating tablet
I20251026 02:56:40.381464 2378 tablet_bootstrap.cc:492] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:40.384533 2378 ts_tablet_manager.cc:1403] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.157s user 0.003s sys 0.007s
I20251026 02:56:40.385353 2378 raft_consensus.cc:359] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.387234 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8a076043f05c42d9afd9699fb4bec5ba
I20251026 02:56:40.387800 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9b9a5daa36b547a7888d38dc3329eaf2
I20251026 02:56:40.387859 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2c3a87b75efb47f28113790cea7ac82d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:40.388046 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e3e8bac83d124aacb292bdec63fbb5fb" } new_table_name: "default.Table_bdf9ad8ed992475dab446da6038be0b3" modify_external_catalogs: true
I20251026 02:56:40.388296 1655 catalog_manager.cc:6719] ReplaceTablet: tablet 9b9a5daa36b547a7888d38dc3329eaf2 deleted and replaced by tablet 6d06944f71fd4e42a6ea46969c86e9f1
W20251026 02:56:40.388518 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c3a87b75efb47f28113790cea7ac82d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.364519 2375 tablet_bootstrap.cc:492] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:40.389906 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a076043f05c42d9afd9699fb4bec5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.390018 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b9a5daa36b547a7888d38dc3329eaf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d06944f71fd4e42a6ea46969c86e9f1) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.390069 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6731d8e782234dea862350a78ebbac32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:40.390156 3547 raft_consensus.cc:493] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.390305 3547 raft_consensus.cc:515] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:40.390654 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6731d8e782234dea862350a78ebbac32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.391137 2375 tablet_bootstrap.cc:654] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.391170 3647 raft_consensus.cc:493] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.391287 3647 raft_consensus.cc:515] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.391492 3547 leader_election.cc:290] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:40.375149 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.375567 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 45e79be9b98945c49fb712019b21b91b with delete_type TABLET_DATA_DELETED (replaced by tablet 1104345f80d6418fa907a8a26924c281) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.376279 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e645650709f47eea4243ab64ec31312 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.392170 3647 leader_election.cc:290] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:40.392308 2372 log.cc:1199] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/44af74033ccf4d048475f12d164938f5
I20251026 02:56:40.392668 2372 ts_tablet_manager.cc:1950] T 44af74033ccf4d048475f12d164938f5 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:40.393235 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f719c720cc474181a63786a3e248ab2c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:40.393424 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 580) took 6844 ms. Trace:
I20251026 02:56:40.393872 1663 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet dba102838945453cbf44e1cfe505a52c (table default.Table_8ffce6cf6b2d44aa8a9459e834de1d80 [id=92310ee99b5c428c9773b0f7a5462a4d]). Expected version 1 got 0
I20251026 02:56:40.393916 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6f9ae327aa4401baf66a737eaa93cf6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:40.394059 2257 raft_consensus.cc:2468] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
W20251026 02:56:40.394578 2053 leader_election.cc:343] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.394933 2053 leader_election.cc:304] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:40.395215 3647 raft_consensus.cc:2804] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:40.395280 3647 raft_consensus.cc:493] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:40.395334 3647 raft_consensus.cc:3060] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.388515 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 99d318aa0d1847c58167bf0af9e75aa2 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:40.398238 3647 raft_consensus.cc:515] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.398553 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 87895fb88c284a4f8d7c58a5ee48ba8e
I20251026 02:56:40.399483 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 87895fb88c284a4f8d7c58a5ee48ba8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.399502 3647 leader_election.cc:290] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:40.399672 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b9a5daa36b547a7888d38dc3329eaf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d06944f71fd4e42a6ea46969c86e9f1) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.401430 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6f9ae327aa4401baf66a737eaa93cf6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:40.401615 2257 raft_consensus.cc:3060] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.404752 2375 tablet_bootstrap.cc:492] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:40.404922 2257 raft_consensus.cc:2468] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:40.405063 2375 ts_tablet_manager.cc:1403] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.002s
I20251026 02:56:40.405401 2053 leader_election.cc:304] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:40.405685 3647 raft_consensus.cc:2804] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:40.405803 3647 raft_consensus.cc:697] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:40.406030 3647 consensus_queue.cc:237] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.407179 2375 raft_consensus.cc:359] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.407541 2375 raft_consensus.cc:385] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.407683 2375 raft_consensus.cc:740] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.408557 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7096b78402074049b9c8ed1b9fe21f0e
I20251026 02:56:40.409576 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 7096b78402074049b9c8ed1b9fe21f0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.409974 2083 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a076043f05c42d9afd9699fb4bec5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.410776 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 87895fb88c284a4f8d7c58a5ee48ba8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.410152 2375 consensus_queue.cc:260] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.411854 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_59ba57e9b91b4b7ca1fefa8880a5cdb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:40.412437 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_000637f9a96344c28b90e88c10c89a07" } new_table_name: "default.Table_b4fba989902f42f189a613083b5c3f74" modify_external_catalogs: true
W20251026 02:56:40.412600 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59ba57e9b91b4b7ca1fefa8880a5cdb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.414039 2381 tablet_bootstrap.cc:492] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:40.414297 2381 ts_tablet_manager.cc:1403] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20251026 02:56:40.414265 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_de99eb535be548db8335e7fbcfe8adb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:40.386656 2376 tablet_replica.cc:333] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:40.388983 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a076043f05c42d9afd9699fb4bec5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.390690 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b9a5daa36b547a7888d38dc3329eaf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d06944f71fd4e42a6ea46969c86e9f1) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.414527 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 7096b78402074049b9c8ed1b9fe21f0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.392760 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6f9ae327aa4401baf66a737eaa93cf6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:40.414832 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de99eb535be548db8335e7fbcfe8adb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:40.415191 2053 leader_election.cc:343] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.414533 2378 raft_consensus.cc:385] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.415534 2381 raft_consensus.cc:359] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.415645 2376 raft_consensus.cc:2243] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:40.414017 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_620d0c4122484e579155fe90d2d5c491" } new_table_name: "default.Table_2a29b2a4df164e5ca2708b4ad13ae9bf" modify_external_catalogs: true
I20251026 02:56:40.415619 2378 raft_consensus.cc:740] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.399354 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6f9ae327aa4401baf66a737eaa93cf6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:40.416373 2381 raft_consensus.cc:385] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.408455 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 741792cf96164695b03de3fb86b12602 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.409154 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet be1c0948c0ae4563b7cb644521914cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.409991 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 87895fb88c284a4f8d7c58a5ee48ba8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.416462 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b83cc3a9c8694aba81d2fcebaaeec62a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:40.416499 2381 raft_consensus.cc:740] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
W20251026 02:56:40.416785 2053 leader_election.cc:343] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:40.416960 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b83cc3a9c8694aba81d2fcebaaeec62a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.416087 2376 raft_consensus.cc:2272] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.417135 2375 ts_tablet_manager.cc:1434] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20251026 02:56:40.417012 2381 consensus_queue.cc:260] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.418047 2376 ts_tablet_manager.cc:1916] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.418200 2381 ts_tablet_manager.cc:1434] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:40.418622 2381 tablet_bootstrap.cc:492] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:40.398200 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f719c720cc474181a63786a3e248ab2c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:40.419450 2053 leader_election.cc:343] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.419564 2053 leader_election.cc:304] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:40.419978 3547 raft_consensus.cc:2749] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:40.418210 2378 consensus_queue.cc:260] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.420590 2381 tablet_bootstrap.cc:654] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.420794 2375 tablet_bootstrap.cc:492] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:40.426743 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 7096b78402074049b9c8ed1b9fe21f0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.433416 1660 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_3af0e3e691254fbb8f124697619830bf [id=31f977e55ba041f59bcc721d469746fd] alter complete (version 3)
I20251026 02:56:40.433748 2378 ts_tablet_manager.cc:1434] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.049s user 0.000s sys 0.002s
I20251026 02:56:40.439675 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet ce922040298d4952af13652eeda7a234 with delete_type TABLET_DATA_DELETED (replaced by tablet e830f1aa1270420c9b28a0491a1a0960) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.439738 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet c31d2704d1f14ddead69c794cc631ec1 with delete_type TABLET_DATA_DELETED (replaced by tablet de9124fe1c8945d9b9565cb732bef20e) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.439486 2378 tablet_bootstrap.cc:492] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:40.439989 2083 tablet_service.cc:1552] Processing DeleteTablet for tablet ec13c5386abe414ba25589e03d5b850a with delete_type TABLET_DATA_DELETED (replaced by tablet 6699b50d87964f6a89c365171253ad25) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.440459 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a3491b303f94fe2a2c838297f73d464 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b5e61afe90e4bc4a2897de21e0b3919) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.446240 2376 ts_tablet_manager.cc:1929] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.448343 2376 log.cc:1199] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/f6d6f024d8b943d5acf69b64bc7c7214
I20251026 02:56:40.449422 2376 ts_tablet_manager.cc:1950] T f6d6f024d8b943d5acf69b64bc7c7214 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
W20251026 02:56:40.451170 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0e6905e5ac147e3bc3a0073816faf76 consensus state has no leader
I20251026 02:56:40.452397 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4236af49f54d455981be3a644c6bb095"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:40.450713 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 666) took 3434 ms. Trace:
I20251026 02:56:40.452593 3547 consensus_queue.cc:1048] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:40.452684 2376 rpcz_store.cc:276] 1026 02:56:37.016003 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.016573 (+ 570us) service_pool.cc:225] Handling call
1026 02:56:40.386345 (+3369772us) ts_tablet_manager.cc:1151] Deleting tablet f6d6f024d8b943d5acf69b64bc7c7214
1026 02:56:40.386354 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.430149 (+ 43795us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.446119 (+ 15970us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.450690 (+ 4571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3369643}
W20251026 02:56:40.453019 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4236af49f54d455981be3a644c6bb095 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.453539 2376 tablet_replica.cc:333] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:40.453730 2376 raft_consensus.cc:2243] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:40.453895 2376 raft_consensus.cc:2272] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.454788 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_efc39d27d9d5438ea04cef669a110c8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.455479 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efc39d27d9d5438ea04cef669a110c8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.455868 2376 ts_tablet_manager.cc:1916] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.460098 2375 tablet_bootstrap.cc:654] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.459981 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e98c5342f720423ba269cf18b16fcfd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.460709 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e98c5342f720423ba269cf18b16fcfd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.463507 3372 raft_consensus.cc:493] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.463650 3372 raft_consensus.cc:515] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.457872 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet f6d6f024d8b943d5acf69b64bc7c7214 (table default.Table_30b96d2d4369458390803c6be18efe0e [id=9fd6e480fc3543149959773e691a3ddd]) successfully deleted
I20251026 02:56:40.464594 3372 leader_election.cc:290] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.465101 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dba102838945453cbf44e1cfe505a52c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:40.465952 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dba102838945453cbf44e1cfe505a52c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:40.466187 3521 consensus_queue.cc:1048] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:40.467417 1915 leader_election.cc:304] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:40.467836 3372 raft_consensus.cc:2749] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:40.468901 2053 consensus_peers.cc:597] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:40.469734 2377 ts_tablet_manager.cc:1929] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.472128 2377 log.cc:1199] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/f85c2745a371455a8dda67f31a19c9d5
I20251026 02:56:40.472692 2377 ts_tablet_manager.cc:1950] T f85c2745a371455a8dda67f31a19c9d5 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:40.473975 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 557) took 7418 ms. Trace:
I20251026 02:56:40.474814 3521 raft_consensus.cc:493] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.475071 3521 raft_consensus.cc:515] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.475690 3521 leader_election.cc:290] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.476702 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30c619ad37364747b6ebf97a73ef45a2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:40.476727 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30c619ad37364747b6ebf97a73ef45a2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:40.476899 2111 raft_consensus.cc:2468] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:40.477402 2194 leader_election.cc:343] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.477814 2195 leader_election.cc:304] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:40.478264 3521 raft_consensus.cc:2804] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:40.478333 3521 raft_consensus.cc:493] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:40.478402 3521 raft_consensus.cc:3060] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.481451 3521 raft_consensus.cc:515] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.482621 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30c619ad37364747b6ebf97a73ef45a2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:40.483223 2194 leader_election.cc:343] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.484164 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30c619ad37364747b6ebf97a73ef45a2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:40.484367 2111 raft_consensus.cc:3060] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.487154 2111 raft_consensus.cc:2468] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:40.489360 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6dbe3f1f3e074da5bae9badcffa01f6b consensus state has no leader
I20251026 02:56:40.491567 2381 tablet_bootstrap.cc:492] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:40.492277 2381 ts_tablet_manager.cc:1403] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.074s user 0.008s sys 0.004s
I20251026 02:56:40.495296 3521 leader_election.cc:290] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.496143 2377 rpcz_store.cc:276] 1026 02:56:33.055871 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.082867 (+ 26996us) service_pool.cc:225] Handling call
1026 02:56:40.351100 (+7268233us) ts_tablet_manager.cc:1151] Deleting tablet f85c2745a371455a8dda67f31a19c9d5
1026 02:56:40.351112 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.422236 (+ 71124us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.469618 (+ 47382us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.473956 (+ 4338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7246363}
I20251026 02:56:40.496937 2377 tablet_replica.cc:333] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:40.497563 2377 raft_consensus.cc:2243] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:40.497802 2195 leader_election.cc:304] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:40.499840 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet f85c2745a371455a8dda67f31a19c9d5 (table default.Table_542a136f8cdd4d32b2e1e6fbf7bcc448 [id=f802cc99a8ec439a85a18e9b9a2826e7]) successfully deleted
I20251026 02:56:40.499330 2381 raft_consensus.cc:359] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.500269 2381 raft_consensus.cc:385] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.500674 2377 raft_consensus.cc:2272] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.500864 2381 raft_consensus.cc:740] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.500322 3239 raft_consensus.cc:2804] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:40.501660 3239 raft_consensus.cc:697] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:40.501869 3239 consensus_queue.cc:237] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.501940 2381 consensus_queue.cc:260] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.505172 2377 ts_tablet_manager.cc:1916] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.514075 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e6acd68207564173ac53389222b0c621" } new_table_name: "default.Table_b425c32631244cb0834828db37fc0c88" modify_external_catalogs: true
I20251026 02:56:40.514014 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9304ae0aeabf48e99b2f10c5fba8cd1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:40.514616 2375 tablet_bootstrap.cc:492] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:40.514760 2375 ts_tablet_manager.cc:1403] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.095s user 0.004s sys 0.001s
I20251026 02:56:40.515542 2375 raft_consensus.cc:359] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.515596 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9b0942af08414e7bb088a14c781f0edd" } new_table_name: "default.Table_4e2e1ce85b8246388602db5ddc857f79" modify_external_catalogs: true
I20251026 02:56:40.515699 2375 raft_consensus.cc:385] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.515770 2375 raft_consensus.cc:740] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.516083 2375 consensus_queue.cc:260] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.517720 2375 ts_tablet_manager.cc:1434] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 02:56:40.518426 2375 tablet_bootstrap.cc:492] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:40.526648 2378 tablet_bootstrap.cc:654] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:40.534870 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9304ae0aeabf48e99b2f10c5fba8cd1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:40.544500 1649 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 6dbe3f1f3e074da5bae9badcffa01f6b (table default.Table_b4fba989902f42f189a613083b5c3f74 [id=c788731642224bcf9c22c007e8ab35c0], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6dbe3f1f3e074da5bae9badcffa01f6b consensus state has no leader
I20251026 02:56:40.505295 2381 ts_tablet_manager.cc:1434] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20251026 02:56:40.552778 2381 tablet_bootstrap.cc:492] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:40.552524 2375 tablet_bootstrap.cc:654] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.554905 1974 raft_consensus.cc:3060] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.561638 3622 consensus_queue.cc:1048] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:40.583038 1982 raft_consensus.cc:3060] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.583995 2381 tablet_bootstrap.cc:654] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.587850 3616 consensus_queue.cc:1048] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:40.588197 2375 tablet_bootstrap.cc:492] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:40.588891 2375 ts_tablet_manager.cc:1403] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.001s
I20251026 02:56:40.596637 3521 consensus_queue.cc:1048] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:40.599068 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_26dd741c83ff419f8c9bc808e83d9c72" } modify_external_catalogs: true
I20251026 02:56:40.599313 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_26dd741c83ff419f8c9bc808e83d9c72" } modify_external_catalogs: true
I20251026 02:56:40.602908 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_dc1b77e5238847adbd80a2df2274c003" } modify_external_catalogs: true
I20251026 02:56:40.603107 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_dc1b77e5238847adbd80a2df2274c003" } modify_external_catalogs: true
I20251026 02:56:40.605553 3697 tablet.cc:1722] T 23fd1284c9cb41bd98209ccbfe342bca P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:40.606099 3696 tablet.cc:1722] T 23fd1284c9cb41bd98209ccbfe342bca P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:40.607640 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e1085a562ca74f1bb56e6661271a3b50" } modify_external_catalogs: true
I20251026 02:56:40.607838 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e1085a562ca74f1bb56e6661271a3b50" } modify_external_catalogs: true
I20251026 02:56:40.609666 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9e3fab0107274734a5532679436e4931"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.610270 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e3fab0107274734a5532679436e4931 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.611308 2027 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:40.603576 2375 raft_consensus.cc:359] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.613185 2375 raft_consensus.cc:385] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.613837 2375 raft_consensus.cc:740] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.615497 2375 consensus_queue.cc:260] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.621351 2375 ts_tablet_manager.cc:1434] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.032s user 0.003s sys 0.000s
I20251026 02:56:40.622526 2375 tablet_bootstrap.cc:492] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:40.631139 3647 raft_consensus.cc:493] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.631443 3647 raft_consensus.cc:515] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:40.632179 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1dec51b88e3545d1a0a7793613540ee3 consensus state has no leader
I20251026 02:56:40.632504 3647 leader_election.cc:290] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:40.632828 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec13c5386abe414ba25589e03d5b850a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:40.632918 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec13c5386abe414ba25589e03d5b850a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:40.633396 2053 leader_election.cc:343] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:40.633747 2053 leader_election.cc:343] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.633836 2053 leader_election.cc:304] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:40.634207 3647 raft_consensus.cc:2749] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:40.633236 2375 tablet_bootstrap.cc:654] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:40.642585 2194 consensus_peers.cc:597] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:40.647547 2377 ts_tablet_manager.cc:1929] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:40.647697 2377 log.cc:1199] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/548a7f9628ac4623b2c2f723f83c1e45
I20251026 02:56:40.648113 2377 ts_tablet_manager.cc:1950] T 548a7f9628ac4623b2c2f723f83c1e45 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:40.648521 2378 tablet_bootstrap.cc:492] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:40.648761 2378 ts_tablet_manager.cc:1403] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.210s user 0.004s sys 0.016s
I20251026 02:56:40.649596 2378 raft_consensus.cc:359] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.649873 2378 raft_consensus.cc:385] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.650017 2378 raft_consensus.cc:740] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.650463 2378 consensus_queue.cc:260] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.651454 2378 ts_tablet_manager.cc:1434] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:40.651564 2376 ts_tablet_manager.cc:1929] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.655227 2381 tablet_bootstrap.cc:492] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:40.655500 2381 ts_tablet_manager.cc:1403] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.103s user 0.003s sys 0.000s
I20251026 02:56:40.655507 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6c5a2f23ffbf41d294fbf17be375e630" } modify_external_catalogs: true
I20251026 02:56:40.655753 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6c5a2f23ffbf41d294fbf17be375e630" } modify_external_catalogs: true
I20251026 02:56:40.662060 2381 raft_consensus.cc:359] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.662873 2376 log.cc:1199] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/1e56b259a314422aba35eb129021bbeb
I20251026 02:56:40.665184 2376 ts_tablet_manager.cc:1950] T 1e56b259a314422aba35eb129021bbeb P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:40.666121 2381 raft_consensus.cc:385] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.666566 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 667) took 3606 ms. Trace:
I20251026 02:56:40.666939 2381 raft_consensus.cc:740] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.667544 2381 consensus_queue.cc:260] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.668251 2376 rpcz_store.cc:276] 1026 02:56:37.060493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.060751 (+ 258us) service_pool.cc:225] Handling call
1026 02:56:40.453486 (+3392735us) ts_tablet_manager.cc:1151] Deleting tablet 1e56b259a314422aba35eb129021bbeb
1026 02:56:40.453497 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.528937 (+ 75440us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.651453 (+122516us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.666547 (+ 15094us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3392545}
I20251026 02:56:40.669591 2381 ts_tablet_manager.cc:1434] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251026 02:56:40.660607 2378 tablet_bootstrap.cc:492] T c4b0933f71cb4ac7a58986d16b613e64 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:40.671804 2378 tablet_bootstrap.cc:654] T c4b0933f71cb4ac7a58986d16b613e64 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:40.673653 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a956ebfb1eaa4d6fb1575b4165293193 consensus state has no leader
I20251026 02:56:40.676290 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 340bf8c78a88413299b61a52f0a578f3
I20251026 02:56:40.677081 2376 tablet_replica.cc:333] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:40.677426 2083 tablet_service.cc:1552] Processing DeleteTablet for tablet 340bf8c78a88413299b61a52f0a578f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.677440 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3f3634e9694e4cafab2335c0fab273cc" } new_table_name: "default.Table_b7613322967f46cb987deee2389e9ce5" modify_external_catalogs: true
I20251026 02:56:40.678079 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 340bf8c78a88413299b61a52f0a578f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.678859 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a036080d1e7141269eec20b1f8e87f9c" } modify_external_catalogs: true
I20251026 02:56:40.679032 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a036080d1e7141269eec20b1f8e87f9c" } modify_external_catalogs: true
I20251026 02:56:40.681217 2376 raft_consensus.cc:2243] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:40.678784 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 340bf8c78a88413299b61a52f0a578f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.680109 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 565) took 7320 ms. Trace:
I20251026 02:56:40.681524 2377 rpcz_store.cc:276] 1026 02:56:33.359770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.361305 (+ 1535us) service_pool.cc:225] Handling call
1026 02:56:40.496884 (+7135579us) ts_tablet_manager.cc:1151] Deleting tablet 548a7f9628ac4623b2c2f723f83c1e45
1026 02:56:40.496895 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.549156 (+ 52261us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.647434 (+ 98278us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.680062 (+ 32628us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7135407}
I20251026 02:56:40.682288 2376 raft_consensus.cc:2272] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.683241 2376 ts_tablet_manager.cc:1916] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.684099 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_64db0f5adb1740ce80b215181ec835fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.684880 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64db0f5adb1740ce80b215181ec835fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.686452 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 1e56b259a314422aba35eb129021bbeb (table default.Table_25607539f64440cb8ff2c24a0a3a3bdd [id=5dbfb5d912ba433baa46bd52bbf6dbee]) successfully deleted
I20251026 02:56:40.687131 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b17021b7e5d845748cfd8d145b9721ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.688009 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b17021b7e5d845748cfd8d145b9721ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.691376 2378 tablet_bootstrap.cc:492] T c4b0933f71cb4ac7a58986d16b613e64 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:40.691571 2378 ts_tablet_manager.cc:1403] T c4b0933f71cb4ac7a58986d16b613e64 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.003s
I20251026 02:56:40.690198 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 548a7f9628ac4623b2c2f723f83c1e45 (table default.Table_f8baefeba2b54cb39affb2d4f1b0bef8 [id=d9a6b0b6f3e246a3aa8fbd1167205264]) successfully deleted
I20251026 02:56:40.692565 2377 tablet_replica.cc:333] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:40.692544 2378 raft_consensus.cc:359] T c4b0933f71cb4ac7a58986d16b613e64 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.692780 2377 raft_consensus.cc:2243] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:40.692788 2381 tablet_bootstrap.cc:492] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:40.692919 2378 raft_consensus.cc:385] T c4b0933f71cb4ac7a58986d16b613e64 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.693169 2377 raft_consensus.cc:2272] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.693754 2377 ts_tablet_manager.cc:1916] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.697484 2372 rpcz_store.cc:276] 1026 02:56:33.548496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.548539 (+ 43us) service_pool.cc:225] Handling call
1026 02:56:40.280680 (+6732141us) ts_tablet_manager.cc:1151] Deleting tablet 44af74033ccf4d048475f12d164938f5
1026 02:56:40.280690 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.336466 (+ 55776us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.360519 (+ 24053us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.393404 (+ 32885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6731994}
I20251026 02:56:40.697683 2381 tablet_bootstrap.cc:654] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.698201 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 44af74033ccf4d048475f12d164938f5 (table default.Table_e4fbc44faf984982986abfb227d6663d [id=b35a8189bbc04e94bb097c9e13a38ec6]) successfully deleted
I20251026 02:56:40.698583 2372 tablet_replica.cc:333] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:40.698858 2372 raft_consensus.cc:2243] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:40.699303 2372 raft_consensus.cc:2272] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.699923 2372 ts_tablet_manager.cc:1916] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.701306 2375 tablet_bootstrap.cc:492] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:40.701555 2375 ts_tablet_manager.cc:1403] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.079s user 0.009s sys 0.000s
I20251026 02:56:40.702732 2375 raft_consensus.cc:359] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.703044 2375 raft_consensus.cc:385] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.703177 2375 raft_consensus.cc:740] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.703606 2375 consensus_queue.cc:260] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.704500 3700 tablet.cc:1722] T 23fd1284c9cb41bd98209ccbfe342bca P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:40.704623 2375 ts_tablet_manager.cc:1434] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:40.705164 2378 raft_consensus.cc:740] T c4b0933f71cb4ac7a58986d16b613e64 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.705874 2375 tablet_bootstrap.cc:492] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:40.705892 2378 consensus_queue.cc:260] T c4b0933f71cb4ac7a58986d16b613e64 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.710081 2378 ts_tablet_manager.cc:1434] T c4b0933f71cb4ac7a58986d16b613e64 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.018s user 0.000s sys 0.001s
I20251026 02:56:40.710785 2375 tablet_bootstrap.cc:654] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.712929 3372 raft_consensus.cc:493] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.713063 3372 raft_consensus.cc:515] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.713702 3372 leader_election.cc:290] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.719051 2107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741792cf96164695b03de3fb86b12602" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:40.725374 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0937c297edcb4210abad0431e5108adb consensus state has no leader
I20251026 02:56:40.726600 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f12577c6a60b4858b2e778692827a023" } new_table_name: "default.Table_dabd227b99964351870151b7731b8175" modify_external_catalogs: true
I20251026 02:56:40.729928 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741792cf96164695b03de3fb86b12602" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:40.731983 1915 leader_election.cc:304] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:40.732848 3372 raft_consensus.cc:2749] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:40.748498 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_279edde3f26a49bcbf0c6c501709fadc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.749189 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_279edde3f26a49bcbf0c6c501709fadc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.752584 3647 raft_consensus.cc:493] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.752768 3647 raft_consensus.cc:515] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.753377 3647 leader_election.cc:290] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:40.763715 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_94329ed6f5fc42e6b4d1de6b1565adc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.764430 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94329ed6f5fc42e6b4d1de6b1565adc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.767503 2165 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:40.767922 2378 tablet_bootstrap.cc:492] T 8f3ae3a377af46449c3faca339f86053 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:40.768146 2053 consensus_peers.cc:597] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:40.768939 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8862cd48a5744c99b32f365cb77fa7c4 consensus state has no leader
I20251026 02:56:40.769410 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ecf77fe2da2f4a9eb37ccdb284cd45af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:40.769688 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62d30d79a6e4432a8dbe40a104e87fa0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:40.769982 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecf77fe2da2f4a9eb37ccdb284cd45af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:40.770534 2053 leader_election.cc:343] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.771760 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:40.780417 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d00f935fbc4b4d80820018fa2857a22d" } new_table_name: "default.Table_8ea5a407ed7440bf8bbe18a66503efed" modify_external_catalogs: true
I20251026 02:56:40.791175 1657 catalog_manager.cc:5649] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:40.791641 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe (table default.Table_47994972d3e0469c9acc96120aad4e1f [id=87349d9c6c764382b828e716e7854c59]). Expected version 3 got 0
I20251026 02:56:40.794379 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 7c132573f9934f629b7be833deb602ee (DEFAULT_TABLE table=default.Table_7e7480c10911493eaab01f1e2b1a88ba [id=48799764ddc74c4fb878c57ff1b7e329]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.794781 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c132573f9934f629b7be833deb602ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.796757 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62d30d79a6e4432a8dbe40a104e87fa0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:40.797156 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 7c132573f9934f629b7be833deb602ee (DEFAULT_TABLE table=default.Table_7e7480c10911493eaab01f1e2b1a88ba [id=48799764ddc74c4fb878c57ff1b7e329]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.797560 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c132573f9934f629b7be833deb602ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.797796 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 863f441021794237927c24c44bed3d70 (DEFAULT_TABLE table=default.Table_ef87e98153774f2b92a5b45f9191806e [id=ca7569a5d28b49668e1231bafa536144]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.798151 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 863f441021794237927c24c44bed3d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.798800 3521 raft_consensus.cc:493] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.798923 3521 raft_consensus.cc:515] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.801240 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff242ef9541a42d793c4484190956da6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:40.805441 2194 leader_election.cc:343] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.807251 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 863f441021794237927c24c44bed3d70 (DEFAULT_TABLE table=default.Table_ef87e98153774f2b92a5b45f9191806e [id=ca7569a5d28b49668e1231bafa536144]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.807708 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 863f441021794237927c24c44bed3d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.807855 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 94d5faa4584a4e36b38bdd552d641994 (DEFAULT_TABLE table=default.Table_3ee83bc1426c40dfb0c28e58715a9476 [id=33d4a00dc6e64b059c9b01c8e91fc654]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.808239 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94d5faa4584a4e36b38bdd552d641994. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.809266 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 7c132573f9934f629b7be833deb602ee (DEFAULT_TABLE table=default.Table_7e7480c10911493eaab01f1e2b1a88ba [id=48799764ddc74c4fb878c57ff1b7e329]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.809670 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c132573f9934f629b7be833deb602ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.812543 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 94d5faa4584a4e36b38bdd552d641994 (DEFAULT_TABLE table=default.Table_3ee83bc1426c40dfb0c28e58715a9476 [id=33d4a00dc6e64b059c9b01c8e91fc654]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.812613 3521 leader_election.cc:290] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.813913 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94d5faa4584a4e36b38bdd552d641994. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.814793 2377 ts_tablet_manager.cc:1929] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:40.814895 2377 log.cc:1199] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/a821ed8a686c452a975acb96b5c74778
I20251026 02:56:40.815224 2377 ts_tablet_manager.cc:1950] T a821ed8a686c452a975acb96b5c74778 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:40.815313 2096 tablet_service.cc:1505] Processing CreateTablet for tablet c2dfd1aac7744404b3e5eedcb13c71be (DEFAULT_TABLE table=default.Table_502d978bc87a4bdcbd6d61a2710dc639 [id=21a1112be0654648b1c829c309ebdb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.815645 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2dfd1aac7744404b3e5eedcb13c71be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.815923 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 567) took 7453 ms. Trace:
I20251026 02:56:40.815999 2377 rpcz_store.cc:276] 1026 02:56:33.362067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.362425 (+ 358us) service_pool.cc:225] Handling call
1026 02:56:40.682062 (+7319637us) ts_tablet_manager.cc:1151] Deleting tablet a821ed8a686c452a975acb96b5c74778
1026 02:56:40.692509 (+ 10447us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.732161 (+ 39652us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.814696 (+ 82535us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.815901 (+ 1205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7319447}
I20251026 02:56:40.816506 1954 tablet_service.cc:1505] Processing CreateTablet for tablet c2dfd1aac7744404b3e5eedcb13c71be (DEFAULT_TABLE table=default.Table_502d978bc87a4bdcbd6d61a2710dc639 [id=21a1112be0654648b1c829c309ebdb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.816890 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 21d5eb56d29940f48db7395645809ba1 (DEFAULT_TABLE table=default.Table_0ce6582d31c14fdfb11fa3342b1981bb [id=82368672f58d42a482f822eb3df5e212]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.817827 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff242ef9541a42d793c4484190956da6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:40.818022 2118 raft_consensus.cc:2468] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:40.818481 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 863f441021794237927c24c44bed3d70 (DEFAULT_TABLE table=default.Table_ef87e98153774f2b92a5b45f9191806e [id=ca7569a5d28b49668e1231bafa536144]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:40.818569 2053 leader_election.cc:343] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.818681 2053 leader_election.cc:304] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:40.818841 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 863f441021794237927c24c44bed3d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.819156 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet a821ed8a686c452a975acb96b5c74778 (table default.Table_016812daf097412081aff793ad75f72a [id=347d712aca4e477c89805846800fb824]) successfully deleted
I20251026 02:56:40.819386 3647 raft_consensus.cc:2749] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:40.818500 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 94d5faa4584a4e36b38bdd552d641994 (DEFAULT_TABLE table=default.Table_3ee83bc1426c40dfb0c28e58715a9476 [id=33d4a00dc6e64b059c9b01c8e91fc654]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.819916 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94d5faa4584a4e36b38bdd552d641994. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.820297 3647 consensus_queue.cc:1048] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:40.821103 2377 tablet_replica.cc:333] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:40.821336 2377 raft_consensus.cc:2243] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:40.821381 2194 consensus_peers.cc:597] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:40.821576 3647 raft_consensus.cc:493] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.821684 3647 raft_consensus.cc:515] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.822199 3647 leader_election.cc:290] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:40.822886 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c31d2704d1f14ddead69c794cc631ec1" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:40.824275 2372 ts_tablet_manager.cc:1929] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.824388 2372 log.cc:1199] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/a378e913112c443db869016e3b98fd0e
I20251026 02:56:40.824846 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9894f9db0eaf46a9b4b04db096784cc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.825429 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9894f9db0eaf46a9b4b04db096784cc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.826092 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2dfd1aac7744404b3e5eedcb13c71be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.827597 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d5eb56d29940f48db7395645809ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.832497 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7175e585b5e549dc89c9f1eebaae5e96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.833102 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7175e585b5e549dc89c9f1eebaae5e96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.834100 2376 ts_tablet_manager.cc:1929] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:40.834214 2376 log.cc:1199] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/f85c2745a371455a8dda67f31a19c9d5
I20251026 02:56:40.834570 2376 ts_tablet_manager.cc:1950] T f85c2745a371455a8dda67f31a19c9d5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:40.834839 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 668) took 3774 ms. Trace:
I20251026 02:56:40.834903 2376 rpcz_store.cc:276] 1026 02:56:37.060587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.060976 (+ 389us) service_pool.cc:225] Handling call
1026 02:56:40.677026 (+3616050us) ts_tablet_manager.cc:1151] Deleting tablet f85c2745a371455a8dda67f31a19c9d5
1026 02:56:40.677038 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.699887 (+ 22849us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.834022 (+134135us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.834824 (+ 802us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3615920}
I20251026 02:56:40.835422 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet f85c2745a371455a8dda67f31a19c9d5 (table default.Table_542a136f8cdd4d32b2e1e6fbf7bcc448 [id=f802cc99a8ec439a85a18e9b9a2826e7]) successfully deleted
I20251026 02:56:40.837071 2372 ts_tablet_manager.cc:1950] T a378e913112c443db869016e3b98fd0e P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:40.837658 2053 leader_election.cc:343] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:40.838153 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 19918cb3b7d0450f9de17026f8ea3295 consensus state has no leader
I20251026 02:56:40.838574 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b6bd6467e2f24020b6204f82698a81ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.839145 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6bd6467e2f24020b6204f82698a81ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.839141 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 671) took 3734 ms. Trace:
I20251026 02:56:40.839236 2376 rpcz_store.cc:276] 1026 02:56:37.104793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.104846 (+ 53us) service_pool.cc:225] Handling call
1026 02:56:40.838989 (+3734143us) ts_tablet_manager.cc:1151] Deleting tablet 1a36b8a4beca483bb853d27961f5281a
1026 02:56:40.839002 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.839108 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3733941}
W20251026 02:56:40.839749 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 1a36b8a4beca483bb853d27961f5281a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a36b8a4beca483bb853d27961f5281a already in progress: creating tablet
I20251026 02:56:40.840179 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 672) took 3719 ms. Trace:
I20251026 02:56:40.840281 2376 rpcz_store.cc:276] 1026 02:56:37.120375 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.120454 (+ 79us) service_pool.cc:225] Handling call
1026 02:56:40.840028 (+3719574us) ts_tablet_manager.cc:1151] Deleting tablet 7e1a45412db1449a99252112d8110a17
1026 02:56:40.840039 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.840158 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3719328}
W20251026 02:56:40.840684 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7e1a45412db1449a99252112d8110a17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1a45412db1449a99252112d8110a17 already in progress: creating tablet
I20251026 02:56:40.842545 2375 tablet_bootstrap.cc:492] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:40.842978 2377 raft_consensus.cc:2272] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.842972 2240 tablet_service.cc:1505] Processing CreateTablet for tablet c2dfd1aac7744404b3e5eedcb13c71be (DEFAULT_TABLE table=default.Table_502d978bc87a4bdcbd6d61a2710dc639 [id=21a1112be0654648b1c829c309ebdb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.843578 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2dfd1aac7744404b3e5eedcb13c71be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.843624 2375 ts_tablet_manager.cc:1403] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.138s user 0.019s sys 0.006s
I20251026 02:56:40.844879 2377 ts_tablet_manager.cc:1916] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.847779 2195 leader_election.cc:304] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:40.848786 3521 consensus_queue.cc:1048] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:40.848856 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 21d5eb56d29940f48db7395645809ba1 (DEFAULT_TABLE table=default.Table_0ce6582d31c14fdfb11fa3342b1981bb [id=82368672f58d42a482f822eb3df5e212]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.849473 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d5eb56d29940f48db7395645809ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.851568 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c31d2704d1f14ddead69c794cc631ec1" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:40.851794 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 6736de2f44364d8093e6b2ae4dfcf83f (DEFAULT_TABLE table=default.Table_731fe2201e1f483c9997355df8e8b672 [id=f893bfe86e054a31823bd03eabecdd79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.852149 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6736de2f44364d8093e6b2ae4dfcf83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.852185 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 6736de2f44364d8093e6b2ae4dfcf83f (DEFAULT_TABLE table=default.Table_731fe2201e1f483c9997355df8e8b672 [id=f893bfe86e054a31823bd03eabecdd79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.852355 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 572afa86823247a8b1744813350c8eff
I20251026 02:56:40.853062 3616 raft_consensus.cc:2804] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:40.853148 3616 raft_consensus.cc:493] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:40.853085 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 21d5eb56d29940f48db7395645809ba1 (DEFAULT_TABLE table=default.Table_0ce6582d31c14fdfb11fa3342b1981bb [id=82368672f58d42a482f822eb3df5e212]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.853204 3616 raft_consensus.cc:3060] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.853490 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d5eb56d29940f48db7395645809ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.854738 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 6736de2f44364d8093e6b2ae4dfcf83f (DEFAULT_TABLE table=default.Table_731fe2201e1f483c9997355df8e8b672 [id=f893bfe86e054a31823bd03eabecdd79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.855194 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 572afa86823247a8b1744813350c8eff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.856381 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 572afa86823247a8b1744813350c8eff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.857190 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b4fba989902f42f189a613083b5c3f74" } new_table_name: "default.Table_9aac712869d44615bd68059415025e22" modify_external_catalogs: true
I20251026 02:56:40.861536 2378 tablet_bootstrap.cc:654] T 8f3ae3a377af46449c3faca339f86053 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.863786 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 584) took 7287 ms. Trace:
I20251026 02:56:40.864248 2372 rpcz_store.cc:276] 1026 02:56:33.575899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.583459 (+ 7560us) service_pool.cc:225] Handling call
1026 02:56:40.698526 (+7115067us) ts_tablet_manager.cc:1151] Deleting tablet a378e913112c443db869016e3b98fd0e
1026 02:56:40.698540 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.703705 (+ 5165us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.824131 (+120426us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.863766 (+ 39635us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7112739}
I20251026 02:56:40.866266 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7c148c74fbcd4d30b9ba0ea059284427
I20251026 02:56:40.866904 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 7f22c6c163c34ed48a0b7eeb5a574e76 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:40.866976 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c148c74fbcd4d30b9ba0ea059284427 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.867753 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet a378e913112c443db869016e3b98fd0e (table default.Table_f2809b812fe14b2d9e44b22bfa68c1f7 [id=f83014c25cfa449d976ead67854d2ca7]) successfully deleted
I20251026 02:56:40.867341 2375 raft_consensus.cc:359] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.868722 2375 raft_consensus.cc:385] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.869845 2372 tablet_replica.cc:333] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:40.869366 2375 raft_consensus.cc:740] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.870260 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 673) took 3738 ms. Trace:
I20251026 02:56:40.870385 2376 rpcz_store.cc:276] 1026 02:56:37.132312 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.132386 (+ 74us) service_pool.cc:225] Handling call
1026 02:56:40.867735 (+3735349us) ts_tablet_manager.cc:1151] Deleting tablet b68388b2fd014dd1a5f6572796cf53a6
1026 02:56:40.867747 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.870236 (+ 2489us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3733823}
W20251026 02:56:40.870857 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b68388b2fd014dd1a5f6572796cf53a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b68388b2fd014dd1a5f6572796cf53a6 already in progress: creating tablet
I20251026 02:56:40.870062 2372 raft_consensus.cc:2243] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:40.871480 2372 raft_consensus.cc:2272] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.869820 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c148c74fbcd4d30b9ba0ea059284427 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.872418 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 674) took 3737 ms. Trace:
I20251026 02:56:40.872512 2376 rpcz_store.cc:276] 1026 02:56:37.135277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.135353 (+ 76us) service_pool.cc:225] Handling call
1026 02:56:40.872269 (+3736916us) ts_tablet_manager.cc:1151] Deleting tablet 81775055d98349f696cdab6171ffd58b
1026 02:56:40.872280 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.872399 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3736680}
W20251026 02:56:40.872951 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 81775055d98349f696cdab6171ffd58b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81775055d98349f696cdab6171ffd58b already in progress: creating tablet
I20251026 02:56:40.871822 2375 consensus_queue.cc:260] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.872001 2372 ts_tablet_manager.cc:1916] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.876386 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 83416598159a4eb598cf91a4a4e6f44a
I20251026 02:56:40.877545 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 83416598159a4eb598cf91a4a4e6f44a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.879829 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 83416598159a4eb598cf91a4a4e6f44a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.880600 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a72c8f4a62014bbeb2bc7cbb63991756" } new_table_name: "default.Table_f2a821bbac24429d942cbb9a0d7e5b08" modify_external_catalogs: true
I20251026 02:56:40.881043 2375 ts_tablet_manager.cc:1434] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.037s user 0.003s sys 0.000s
I20251026 02:56:40.881685 2375 tablet_bootstrap.cc:492] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:40.888907 2372 ts_tablet_manager.cc:1929] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:40.889596 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4c19009b9b1a44acad306f3b96488546"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:40.891399 2372 log.cc:1199] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/ed836bb7519e4b0290e6547fa57128bc
I20251026 02:56:40.892283 1663 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 4db275efdd344817b11a2168675cf95e (table default.Table_c30cfa52b6524323bda478613d74178b [id=1fd74186ea10498599435dccd7330d25]). Expected version 1 got 0
I20251026 02:56:40.894202 1663 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet c4b0933f71cb4ac7a58986d16b613e64 (table default.Table_e6b19dda39ad45baae642c27b07db42d [id=cba6d4517d244d7580e525729e871745]). Expected version 4 got 0
I20251026 02:56:40.897294 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet a378e913112c443db869016e3b98fd0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.899159 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_111996853fae40db821e68942c9fae5e" } modify_external_catalogs: true
I20251026 02:56:40.899550 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_111996853fae40db821e68942c9fae5e" } modify_external_catalogs: true
I20251026 02:56:40.899633 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 675) took 3747 ms. Trace:
I20251026 02:56:40.899845 2376 rpcz_store.cc:276] 1026 02:56:37.152542 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.157278 (+ 4736us) service_pool.cc:225] Handling call
1026 02:56:40.875216 (+3717938us) ts_tablet_manager.cc:1151] Deleting tablet e32a1f8dcad4430eb35d388bfad7d110
1026 02:56:40.875226 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.899583 (+ 24357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3717712}
W20251026 02:56:40.899817 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c19009b9b1a44acad306f3b96488546 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.900974 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6736de2f44364d8093e6b2ae4dfcf83f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:40.903178 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet e32a1f8dcad4430eb35d388bfad7d110 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e32a1f8dcad4430eb35d388bfad7d110 already in progress: creating tablet
I20251026 02:56:40.905532 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9d12453c4225455881336b9f31d8e378
I20251026 02:56:40.907548 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_645745dc5d2840adaede1a3708978a74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.908249 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_645745dc5d2840adaede1a3708978a74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.909647 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 676) took 3756 ms. Trace:
I20251026 02:56:40.909860 2376 rpcz_store.cc:276] 1026 02:56:37.152975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.158176 (+ 5201us) service_pool.cc:225] Handling call
1026 02:56:40.900443 (+3742267us) ts_tablet_manager.cc:1151] Deleting tablet e830f1aa1270420c9b28a0491a1a0960
1026 02:56:40.909346 (+ 8903us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.909583 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3741931}
I20251026 02:56:40.910502 2376 tablet_replica.cc:333] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:40.910874 2376 raft_consensus.cc:2243] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:40.911249 2376 raft_consensus.cc:2272] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.911705 2083 tablet_service.cc:1552] Processing DeleteTablet for tablet a012d868a78f4324a92db83c81931b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.912537 2376 ts_tablet_manager.cc:1916] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.912634 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d12453c4225455881336b9f31d8e378 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.913529 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet e9cd10092aac40068a1c7f85e514fd37 (table default.Table_fe09be2527a840b09e3e64c27965845d [id=de5272bcc29644208ddd95bdec21254e]). Expected version 1 got 0
I20251026 02:56:40.913858 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (table default.Table_30611e161c274e8b99bd1251e5e527aa [id=b5d91d3e8a2643c09fee85976972391a]). Expected version 1 got 0
I20251026 02:56:40.915158 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d12453c4225455881336b9f31d8e378 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.915530 2375 tablet_bootstrap.cc:654] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.915557 1658 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet c4b0933f71cb4ac7a58986d16b613e64 (table default.Table_e6b19dda39ad45baae642c27b07db42d [id=cba6d4517d244d7580e525729e871745]). Expected version 4 got 0
I20251026 02:56:40.916675 2378 tablet_bootstrap.cc:492] T 8f3ae3a377af46449c3faca339f86053 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:40.916882 2378 ts_tablet_manager.cc:1403] T 8f3ae3a377af46449c3faca339f86053 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.149s user 0.002s sys 0.004s
I20251026 02:56:40.917711 2378 raft_consensus.cc:359] T 8f3ae3a377af46449c3faca339f86053 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.917979 2378 raft_consensus.cc:385] T 8f3ae3a377af46449c3faca339f86053 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.918092 2378 raft_consensus.cc:740] T 8f3ae3a377af46449c3faca339f86053 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.917982 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 426a9e850b7f43e49a5b662a884b1876 (DEFAULT_TABLE table=default.Table_c61f3600fe8e4e81af872d53e8973f25 [id=2dcd76e4feec4ec6944ba529599c6497]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.918413 2378 consensus_queue.cc:260] T 8f3ae3a377af46449c3faca339f86053 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.918646 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426a9e850b7f43e49a5b662a884b1876. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.921775 2372 ts_tablet_manager.cc:1950] T ed836bb7519e4b0290e6547fa57128bc P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:40.922169 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 587) took 7328 ms. Trace:
I20251026 02:56:40.922260 2372 rpcz_store.cc:276] 1026 02:56:33.593568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.593617 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:40.869795 (+7276178us) ts_tablet_manager.cc:1151] Deleting tablet ed836bb7519e4b0290e6547fa57128bc
1026 02:56:40.869807 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.882830 (+ 13023us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.888792 (+ 5962us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:40.922147 (+ 33355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7276004}
I20251026 02:56:40.922713 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet ed836bb7519e4b0290e6547fa57128bc (table default.Table_ec68beb934274468a4f028b3dbaf2cc0 [id=658f4d8e944a4221a722376dc2137896]) successfully deleted
I20251026 02:56:40.923318 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 426a9e850b7f43e49a5b662a884b1876 (DEFAULT_TABLE table=default.Table_c61f3600fe8e4e81af872d53e8973f25 [id=2dcd76e4feec4ec6944ba529599c6497]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:40.923728 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426a9e850b7f43e49a5b662a884b1876. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:40.925762 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6736de2f44364d8093e6b2ae4dfcf83f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:40.928175 2053 leader_election.cc:343] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.928339 2053 leader_election.cc:304] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:40.928794 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet 572afa86823247a8b1744813350c8eff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.929001 2227 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c148c74fbcd4d30b9ba0ea059284427 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.929136 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet a6f9ae327aa4401baf66a737eaa93cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.929366 3647 raft_consensus.cc:2749] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:40.930899 3517 raft_consensus.cc:493] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.931068 2378 ts_tablet_manager.cc:1434] T 8f3ae3a377af46449c3faca339f86053 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20251026 02:56:40.931025 3517 raft_consensus.cc:515] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:40.918118 3622 raft_consensus.cc:493] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.931559 3622 raft_consensus.cc:515] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.933882 2372 tablet_replica.cc:333] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:40.933954 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4d51b54e2a28409fac865e70f2444c78" } new_table_name: "default.Table_c17caa9e7ba04ae2a7d056d8fa619a10" modify_external_catalogs: true
I20251026 02:56:40.934149 2372 raft_consensus.cc:2243] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:40.934484 2372 raft_consensus.cc:2272] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:40.935083 2372 ts_tablet_manager.cc:1916] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:40.935774 2378 tablet_bootstrap.cc:492] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:40.936906 3647 raft_consensus.cc:493] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:40.937037 3647 raft_consensus.cc:515] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.937623 3647 leader_election.cc:290] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:40.939311 2375 tablet_bootstrap.cc:492] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:40.939532 2375 ts_tablet_manager.cc:1403] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20251026 02:56:40.940407 2375 raft_consensus.cc:359] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.940670 2375 raft_consensus.cc:385] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:40.940805 2375 raft_consensus.cc:740] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:40.941210 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 83416598159a4eb598cf91a4a4e6f44a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.941282 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8862cd48a5744c99b32f365cb77fa7c4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:40.941434 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d12453c4225455881336b9f31d8e378 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:40.941488 2375 consensus_queue.cc:260] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.942296 3622 leader_election.cc:290] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:40.942816 2053 leader_election.cc:343] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.943897 2375 ts_tablet_manager.cc:1434] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:40.944176 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882fe81da21344ed99ea735fa79f73c0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:40.944459 2118 raft_consensus.cc:2468] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:40.945894 2375 tablet_bootstrap.cc:492] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:40.946254 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 426a9e850b7f43e49a5b662a884b1876 (DEFAULT_TABLE table=default.Table_c61f3600fe8e4e81af872d53e8973f25 [id=2dcd76e4feec4ec6944ba529599c6497]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:40.948120 2194 consensus_peers.cc:597] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:40.949388 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882fe81da21344ed99ea735fa79f73c0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:40.949779 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfc3aed64ad0439cb22fd6815c973d87" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:40.949967 1980 raft_consensus.cc:2468] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:40.950611 2194 leader_election.cc:343] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.950860 2194 leader_election.cc:304] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:40.955770 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a636a385c0844582a777f2be1f382310"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:40.957115 3517 leader_election.cc:290] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.957690 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426a9e850b7f43e49a5b662a884b1876. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:40.957660 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a636a385c0844582a777f2be1f382310 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.958675 3239 raft_consensus.cc:2804] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 02:56:40.959443 3239 raft_consensus.cc:493] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:40.959848 3239 raft_consensus.cc:3060] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Advancing to term 2
W20251026 02:56:40.959491 2194 consensus_peers.cc:597] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:40.962642 2378 tablet_bootstrap.cc:654] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:40.962778 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8862cd48a5744c99b32f365cb77fa7c4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:40.962980 2259 raft_consensus.cc:2468] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
W20251026 02:56:40.968425 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6dbe3f1f3e074da5bae9badcffa01f6b consensus state has no leader
I20251026 02:56:40.971684 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b3f9a7b483b84dd0a53864640ba2b8b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:40.972630 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3f9a7b483b84dd0a53864640ba2b8b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:40.974767 3616 raft_consensus.cc:515] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:40.976254 3616 leader_election.cc:290] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:40.977340 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff242ef9541a42d793c4484190956da6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:40.977823 1982 raft_consensus.cc:3060] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:40.980196 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7f22c6c163c34ed48a0b7eeb5a574e76
I20251026 02:56:40.980767 1655 catalog_manager.cc:6719] ReplaceTablet: tablet 7f22c6c163c34ed48a0b7eeb5a574e76 deleted and replaced by tablet e4c149e8b7f9420c8dba56da2e21f3f9
W20251026 02:56:40.980937 2194 leader_election.cc:343] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:40.980944 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f22c6c163c34ed48a0b7eeb5a574e76 with delete_type TABLET_DATA_DELETED (replaced by tablet e4c149e8b7f9420c8dba56da2e21f3f9) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:40.983991 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f22c6c163c34ed48a0b7eeb5a574e76 with delete_type TABLET_DATA_DELETED (replaced by tablet e4c149e8b7f9420c8dba56da2e21f3f9) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:40.985162 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_40efa920a1f94812ab827515fddc0033" } new_table_name: "default.Table_555ef24f609a48feaeb346a1523c97d0" modify_external_catalogs: true
W20251026 02:56:40.995669 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28dfe285139042349b5b9d6cbc59a778 consensus state has no leader
I20251026 02:56:41.000253 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_52557bc43e124635b11e4e542e6f62df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.000967 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52557bc43e124635b11e4e542e6f62df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.006597 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 9b976c87b5d04f9294add7761a2afbfd (DEFAULT_TABLE table=default.Table_fef50bc3bf28484f9b1c67ab84027952 [id=b1c4c87e547e43449478937b00705fd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.007057 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b976c87b5d04f9294add7761a2afbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.011602 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a32b1a6b856c47b993a308db94dac357
I20251026 02:56:41.012408 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet a32b1a6b856c47b993a308db94dac357 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:41.013823 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet a32b1a6b856c47b993a308db94dac357 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:41.017315 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet c4b0933f71cb4ac7a58986d16b613e64 (table default.Table_e6b19dda39ad45baae642c27b07db42d [id=cba6d4517d244d7580e525729e871745]). Expected version 4 got 0
I20251026 02:56:41.017362 3239 raft_consensus.cc:515] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.017647 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 8862cd48a5744c99b32f365cb77fa7c4 (table default.Table_dabd227b99964351870151b7731b8175 [id=604570227c014f66aeae89d93edaaafe]). Expected version 1 got 0
I20251026 02:56:41.018488 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfc3aed64ad0439cb22fd6815c973d87" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:41.018702 1980 raft_consensus.cc:3060] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:41.021523 1980 raft_consensus.cc:2468] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 2.
I20251026 02:56:41.022344 2194 leader_election.cc:304] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:41.023149 3616 raft_consensus.cc:2804] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 02:56:41.023336 3616 raft_consensus.cc:697] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 2 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:41.023896 3616 consensus_queue.cc:237] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.025619 2375 tablet_bootstrap.cc:654] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.029865 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 9b976c87b5d04f9294add7761a2afbfd (DEFAULT_TABLE table=default.Table_fef50bc3bf28484f9b1c67ab84027952 [id=b1c4c87e547e43449478937b00705fd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.029865 1946 tablet_service.cc:1505] Processing CreateTablet for tablet dcbe9c57f8224c5c92f82904e97f68e5 (DEFAULT_TABLE table=default.Table_b68d07c85de14db79377412aa4474098 [id=8cbd355f5faf4508ab6d77691f8f09e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.030272 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b976c87b5d04f9294add7761a2afbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.030272 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcbe9c57f8224c5c92f82904e97f68e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.035562 2381 tablet_bootstrap.cc:492] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:41.035784 2381 ts_tablet_manager.cc:1403] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.343s user 0.028s sys 0.042s
W20251026 02:56:41.036082 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 90397.5us
I20251026 02:56:41.036006 3239 leader_election.cc:290] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:41.036794 2195 leader_election.cc:304] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.037612 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9313700399784167b4022a0e6e3a56f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.038331 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9313700399784167b4022a0e6e3a56f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.038637 3239 raft_consensus.cc:2804] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:41.038708 3239 raft_consensus.cc:493] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:41.038765 3239 raft_consensus.cc:3060] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.040488 2053 leader_election.cc:304] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.041049 2377 ts_tablet_manager.cc:1929] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:41.041038 3647 raft_consensus.cc:2804] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:41.041133 3647 raft_consensus.cc:493] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:41.041164 2377 log.cc:1199] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/903e65fa8c214dc9a50cb97cab101d22
I20251026 02:56:41.041205 3647 raft_consensus.cc:3060] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.041502 2377 ts_tablet_manager.cc:1950] T 903e65fa8c214dc9a50cb97cab101d22 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:41.041713 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff242ef9541a42d793c4484190956da6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:41.041927 2118 raft_consensus.cc:3060] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.042246 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 566) took 7682 ms. Trace:
I20251026 02:56:41.042346 2377 rpcz_store.cc:276] 1026 02:56:33.359886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.362795 (+ 2909us) service_pool.cc:225] Handling call
1026 02:56:40.821051 (+7458256us) ts_tablet_manager.cc:1151] Deleting tablet 903e65fa8c214dc9a50cb97cab101d22
1026 02:56:40.821059 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.036010 (+214951us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.040950 (+ 4940us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.042227 (+ 1277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7458071}
I20251026 02:56:41.042716 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 568) took 7578 ms. Trace:
I20251026 02:56:41.042786 2377 rpcz_store.cc:276] 1026 02:56:33.463930 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.463988 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:41.042602 (+7578614us) ts_tablet_manager.cc:1151] Deleting tablet 1fb2ff87cf0248a0a4a77f2733ef867b
1026 02:56:41.042612 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.042701 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7578382}
I20251026 02:56:41.037245 2381 raft_consensus.cc:359] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.043308 2381 raft_consensus.cc:385] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.043376 2381 raft_consensus.cc:740] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.043597 2381 consensus_queue.cc:260] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.044190 3647 raft_consensus.cc:515] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.044832 3647 leader_election.cc:290] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.045188 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8862cd48a5744c99b32f365cb77fa7c4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:41.045722 2053 leader_election.cc:343] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.047546 2118 raft_consensus.cc:2468] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:41.048799 2381 ts_tablet_manager.cc:1434] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20251026 02:56:41.049095 2381 tablet_bootstrap.cc:492] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:41.049249 1946 tablet_service.cc:1505] Processing CreateTablet for tablet f0bb16b00e5344cd9c2d2c2da78b5de5 (DEFAULT_TABLE table=default.Table_38943948af334d7b907bfbd6cd428fd3 [id=737791e9acb74878a57d479a214b9432]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.053072 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0bb16b00e5344cd9c2d2c2da78b5de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.058238 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f22c6c163c34ed48a0b7eeb5a574e76 with delete_type TABLET_DATA_DELETED (replaced by tablet e4c149e8b7f9420c8dba56da2e21f3f9) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.058727 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet a32b1a6b856c47b993a308db94dac357 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.059476 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 9b976c87b5d04f9294add7761a2afbfd (DEFAULT_TABLE table=default.Table_fef50bc3bf28484f9b1c67ab84027952 [id=b1c4c87e547e43449478937b00705fd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.059883 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b976c87b5d04f9294add7761a2afbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.060097 2375 tablet_bootstrap.cc:492] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:41.060261 2375 ts_tablet_manager.cc:1403] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.000s
I20251026 02:56:41.060914 2376 ts_tablet_manager.cc:1929] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.060981 2375 raft_consensus.cc:359] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.061182 2375 raft_consensus.cc:385] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.061230 2375 raft_consensus.cc:740] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.061367 2376 log.cc:1199] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/634d913c98c842a49d97a54f23752d26
I20251026 02:56:41.061417 2375 consensus_queue.cc:260] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.062106 2376 ts_tablet_manager.cc:1950] T 634d913c98c842a49d97a54f23752d26 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:41.063277 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 677) took 3904 ms. Trace:
I20251026 02:56:41.063582 2376 rpcz_store.cc:276] 1026 02:56:37.158345 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.162084 (+ 3739us) service_pool.cc:225] Handling call
1026 02:56:40.910434 (+3748350us) ts_tablet_manager.cc:1151] Deleting tablet 634d913c98c842a49d97a54f23752d26
1026 02:56:40.910445 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.996081 (+ 85636us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.060726 (+ 64645us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.063174 (+ 2448us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3748072}
I20251026 02:56:41.064350 2375 ts_tablet_manager.cc:1434] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:41.064520 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 570) took 7591 ms. Trace:
I20251026 02:56:41.064622 2377 rpcz_store.cc:276] 1026 02:56:33.472882 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.472947 (+ 65us) service_pool.cc:225] Handling call
1026 02:56:41.042982 (+7570035us) ts_tablet_manager.cc:1151] Deleting tablet aa399e6407fd4c4d9871d3d5ccf894ec
1026 02:56:41.042990 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.064492 (+ 21502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7569806}
I20251026 02:56:41.064972 2377 tablet_replica.cc:333] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:41.065187 2377 raft_consensus.cc:2243] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:41.065886 2375 tablet_bootstrap.cc:492] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:41.066411 2233 tablet_service.cc:1505] Processing CreateTablet for tablet dcbe9c57f8224c5c92f82904e97f68e5 (DEFAULT_TABLE table=default.Table_b68d07c85de14db79377412aa4474098 [id=8cbd355f5faf4508ab6d77691f8f09e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.066790 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcbe9c57f8224c5c92f82904e97f68e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.067225 2377 raft_consensus.cc:2272] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.067770 2375 tablet_bootstrap.cc:654] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.068143 2372 ts_tablet_manager.cc:1929] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:41.068308 2372 log.cc:1199] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/049a915b35224cdaa38df77c734a0f01
I20251026 02:56:41.068343 2377 ts_tablet_manager.cc:1916] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.068758 2372 ts_tablet_manager.cc:1950] T 049a915b35224cdaa38df77c734a0f01 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:41.069820 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8862cd48a5744c99b32f365cb77fa7c4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:41.069995 2259 raft_consensus.cc:3060] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.073964 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 592) took 7443 ms. Trace:
I20251026 02:56:41.074133 2372 rpcz_store.cc:276] 1026 02:56:33.630432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.630681 (+ 249us) service_pool.cc:225] Handling call
1026 02:56:40.933828 (+7303147us) ts_tablet_manager.cc:1151] Deleting tablet 049a915b35224cdaa38df77c734a0f01
1026 02:56:40.933840 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:40.940061 (+ 6221us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.068021 (+127960us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.073937 (+ 5916us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49,"tablet-delete.queue_time_us":7302961}
I20251026 02:56:41.074579 2372 tablet_replica.cc:333] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:41.075155 2372 raft_consensus.cc:2243] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.075840 2372 raft_consensus.cc:2272] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.076735 2372 ts_tablet_manager.cc:1916] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.078276 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 684) took 3702 ms. Trace:
I20251026 02:56:41.078660 2259 raft_consensus.cc:2468] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:41.078891 2376 rpcz_store.cc:276] 1026 02:56:37.363027 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.363085 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:41.065147 (+3702062us) ts_tablet_manager.cc:1151] Deleting tablet b8cad7adae7e43469ceadbedeee53564
1026 02:56:41.065158 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.065485 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3701815}
W20251026 02:56:41.080821 2195 leader_election.cc:343] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: cfc3aed64ad0439cb22fd6815c973d87
I20251026 02:56:41.081655 2195 leader_election.cc:304] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.082167 3239 raft_consensus.cc:515] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.082824 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 685) took 3718 ms. Trace:
I20251026 02:56:41.083499 3239 leader_election.cc:290] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:41.083948 3239 raft_consensus.cc:2804] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:41.084079 3239 raft_consensus.cc:697] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:41.084625 3239 consensus_queue.cc:237] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.084257 2376 rpcz_store.cc:276] 1026 02:56:37.364563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.364611 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:41.081043 (+3716432us) ts_tablet_manager.cc:1151] Deleting tablet ba08469339184b11b9ced9e6bdd63d42
1026 02:56:41.081301 (+ 258us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.082678 (+ 1377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3715907}
I20251026 02:56:41.088817 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 903e65fa8c214dc9a50cb97cab101d22 (table default.Table_2d168c00cb884360bdb2475bd391f576 [id=65b5770c4b9243b892bfab963a439bb9]) successfully deleted
I20251026 02:56:41.088699 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2811eabdf1ec4f61b92ab33a8c7ec6c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.089422 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 1fb2ff87cf0248a0a4a77f2733ef867b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fb2ff87cf0248a0a4a77f2733ef867b already in progress: creating tablet
W20251026 02:56:41.089766 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet aa399e6407fd4c4d9871d3d5ccf894ec because the tablet was not found. No further retry: Not found: Tablet not found: aa399e6407fd4c4d9871d3d5ccf894ec
I20251026 02:56:41.089740 3647 raft_consensus.cc:493] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.089915 3647 raft_consensus.cc:515] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.090080 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 634d913c98c842a49d97a54f23752d26 (table default.Table_d2dcec579c684b92a92d98fb59ee3da1 [id=4f7edfb6d75c4a0b94ca99bb9cfe14f3]) successfully deleted
W20251026 02:56:41.090306 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b8cad7adae7e43469ceadbedeee53564 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8cad7adae7e43469ceadbedeee53564 already in progress: creating tablet
W20251026 02:56:41.090548 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ba08469339184b11b9ced9e6bdd63d42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba08469339184b11b9ced9e6bdd63d42 already in progress: creating tablet
I20251026 02:56:41.090706 3647 leader_election.cc:290] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.091060 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 049a915b35224cdaa38df77c734a0f01 (table default.Table_1818605e0bc548938337de5d5fc7a781 [id=f1f9d2ad39b34abbbe989bc3fac62b6c]) successfully deleted
W20251026 02:56:41.091742 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2811eabdf1ec4f61b92ab33a8c7ec6c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.095973 1945 tablet_service.cc:1505] Processing CreateTablet for tablet abf58185766448d1a7c4f3d304b0c5b1 (DEFAULT_TABLE table=default.Table_0b10d09d420e46ac9a13bada7fd2f740 [id=0c49b29c6a764cbe91959c5909e09496]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.096724 2083 tablet_service.cc:1505] Processing CreateTablet for tablet abf58185766448d1a7c4f3d304b0c5b1 (DEFAULT_TABLE table=default.Table_0b10d09d420e46ac9a13bada7fd2f740 [id=0c49b29c6a764cbe91959c5909e09496]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:41.096757 2195 leader_election.cc:343] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: cfc3aed64ad0439cb22fd6815c973d87
I20251026 02:56:41.097133 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf58185766448d1a7c4f3d304b0c5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.097653 1658 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_2a29b2a4df164e5ca2708b4ad13ae9bf [id=6734e5731cc94e71b43c76512f649d57] alter complete (version 3)
I20251026 02:56:41.100173 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet dbf2a1ce78154415b23ab6d0efbc1f52 with delete_type TABLET_DATA_DELETED (replaced by tablet f51f8fef342b46abafff5bf12dba7184) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:41.101040 2053 leader_election.cc:304] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.101552 3647 raft_consensus.cc:2804] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:41.101692 3647 raft_consensus.cc:697] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:41.101999 3647 consensus_queue.cc:237] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.103477 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 882fe81da21344ed99ea735fa79f73c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.103502 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882fe81da21344ed99ea735fa79f73c0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:41.103662 2111 raft_consensus.cc:3060] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.103615 2242 tablet_service.cc:1505] Processing CreateTablet for tablet f0bb16b00e5344cd9c2d2c2da78b5de5 (DEFAULT_TABLE table=default.Table_38943948af334d7b907bfbd6cd428fd3 [id=737791e9acb74878a57d479a214b9432]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.103989 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0bb16b00e5344cd9c2d2c2da78b5de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.104151 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 049a915b35224cdaa38df77c734a0f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:41.104363 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c1e63ac8c5645f3ba1578792c0a0dae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:41.105618 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_17551d2211c14255842b226a8b8fcdd1" } new_table_name: "default.Table_aeb8c52cba3e49768ced34f8a8b72298" modify_external_catalogs: true
I20251026 02:56:41.106232 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8ffce6cf6b2d44aa8a9459e834de1d80" } modify_external_catalogs: true
I20251026 02:56:41.106431 2111 raft_consensus.cc:2468] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:41.106453 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8ffce6cf6b2d44aa8a9459e834de1d80" } modify_external_catalogs: true
I20251026 02:56:41.107126 2231 tablet_service.cc:1505] Processing CreateTablet for tablet abf58185766448d1a7c4f3d304b0c5b1 (DEFAULT_TABLE table=default.Table_0b10d09d420e46ac9a13bada7fd2f740 [id=0c49b29c6a764cbe91959c5909e09496]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.107486 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf58185766448d1a7c4f3d304b0c5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.107992 2195 leader_election.cc:304] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:41.108346 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c119e6aabe874f9495d15c66088c7440" } modify_external_catalogs: true
I20251026 02:56:41.108561 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c119e6aabe874f9495d15c66088c7440" } modify_external_catalogs: true
I20251026 02:56:41.109160 3537 raft_consensus.cc:2804] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:41.109267 3537 raft_consensus.cc:697] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:41.109516 3537 consensus_queue.cc:237] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.115074 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48105e54d6324622a032e1288a46188c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:41.115629 2053 leader_election.cc:343] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.118733 1978 raft_consensus.cc:3060] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.120653 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48105e54d6324622a032e1288a46188c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:41.121023 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 686) took 3746 ms. Trace:
I20251026 02:56:41.121114 2376 rpcz_store.cc:276] 1026 02:56:37.375025 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.375086 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:41.087790 (+3712704us) ts_tablet_manager.cc:1151] Deleting tablet 6108219254bd448c852096c0c967384b
1026 02:56:41.118134 (+ 30344us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.121004 (+ 2870us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3712110}
W20251026 02:56:41.121587 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 6108219254bd448c852096c0c967384b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6108219254bd448c852096c0c967384b already in progress: creating tablet
W20251026 02:56:41.122561 2053 leader_election.cc:343] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.122677 2053 leader_election.cc:304] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.123157 3643 raft_consensus.cc:2749] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:41.127125 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 109073eb664d4090b7a2b786146e13fa consensus state has no leader
I20251026 02:56:41.129077 1659 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 8f3ae3a377af46449c3faca339f86053 (table default.Table_3af0e3e691254fbb8f124697619830bf [id=31f977e55ba041f59bcc721d469746fd]). Expected version 3 got 0
I20251026 02:56:41.130553 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6cd4ee82e9ec406f971669b7f0722677" } modify_external_catalogs: true
I20251026 02:56:41.130681 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf58185766448d1a7c4f3d304b0c5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.130756 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6cd4ee82e9ec406f971669b7f0722677" } modify_external_catalogs: true
I20251026 02:56:41.131212 2082 tablet_service.cc:1505] Processing CreateTablet for tablet f0bb16b00e5344cd9c2d2c2da78b5de5 (DEFAULT_TABLE table=default.Table_38943948af334d7b907bfbd6cd428fd3 [id=737791e9acb74878a57d479a214b9432]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.131604 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0bb16b00e5344cd9c2d2c2da78b5de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.132401 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_173c6dd0f5b241c8a5ce1e918bde90a2" } modify_external_catalogs: true
I20251026 02:56:41.132427 3712 consensus_queue.cc:1048] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:41.132571 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_173c6dd0f5b241c8a5ce1e918bde90a2" } modify_external_catalogs: true
I20251026 02:56:41.136247 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882fe81da21344ed99ea735fa79f73c0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:41.136142 1659 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 386e1bd4635b4a7e8e7f249b577fd978 (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]). Expected version 2 got 0
W20251026 02:56:41.137192 2194 leader_election.cc:343] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:41.137602 2194 consensus_peers.cc:597] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:41.137980 2096 tablet_service.cc:1505] Processing CreateTablet for tablet dcbe9c57f8224c5c92f82904e97f68e5 (DEFAULT_TABLE table=default.Table_b68d07c85de14db79377412aa4474098 [id=8cbd355f5faf4508ab6d77691f8f09e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.138448 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcbe9c57f8224c5c92f82904e97f68e5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:41.138811 1659 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 386e1bd4635b4a7e8e7f249b577fd978 (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 386e1bd4635b4a7e8e7f249b577fd978 consensus state has no leader
I20251026 02:56:41.139673 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2cfec52f9de94b2096a85e6c32337bcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.140367 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cfec52f9de94b2096a85e6c32337bcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.142175 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 688) took 3714 ms. Trace:
I20251026 02:56:41.142309 2376 rpcz_store.cc:276] 1026 02:56:37.427794 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.429012 (+ 1218us) service_pool.cc:225] Handling call
1026 02:56:41.121379 (+3692367us) ts_tablet_manager.cc:1151] Deleting tablet 7f5cd37c87f34a40bf05829af6de0473
1026 02:56:41.142021 (+ 20642us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.142141 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3691618}
I20251026 02:56:41.144438 2377 ts_tablet_manager.cc:1929] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251026 02:56:41.147135 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e1291f47f384ff38eadd16040db65b8 consensus state has no leader
I20251026 02:56:41.151968 2083 tablet_service.cc:1552] Processing DeleteTablet for tablet 86e483c52e514f1da54335479f59d68e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:41.153951 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_44584371880e4feeb01066c835d98432"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.154729 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44584371880e4feeb01066c835d98432 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.159310 2378 tablet_bootstrap.cc:492] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:41.159559 2378 ts_tablet_manager.cc:1403] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.224s user 0.008s sys 0.004s
I20251026 02:56:41.160547 2378 raft_consensus.cc:359] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.160754 2378 raft_consensus.cc:385] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.160825 2378 raft_consensus.cc:740] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.161072 2378 consensus_queue.cc:260] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.161717 2378 ts_tablet_manager.cc:1434] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:41.165091 2377 log.cc:1199] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/634d913c98c842a49d97a54f23752d26
I20251026 02:56:41.165602 2377 ts_tablet_manager.cc:1950] T 634d913c98c842a49d97a54f23752d26 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:41.171119 2378 tablet_bootstrap.cc:492] T 046a34f9408b406ba08471421f422064 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:41.174228 2375 tablet_bootstrap.cc:492] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:41.179754 3712 consensus_queue.cc:1048] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:41.181465 3587 raft_consensus.cc:493] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.186774 3587 raft_consensus.cc:515] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.187505 3587 leader_election.cc:290] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.188242 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c6e3e47b3e7421e945cfbe671b06abe" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:41.188504 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c6e3e47b3e7421e945cfbe671b06abe" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:41.188755 2053 leader_election.cc:343] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:41.188954 2053 leader_election.cc:343] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.189033 2053 leader_election.cc:304] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.189412 3587 raft_consensus.cc:2749] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:41.189718 2375 ts_tablet_manager.cc:1403] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.124s user 0.005s sys 0.000s
I20251026 02:56:41.190518 2375 raft_consensus.cc:359] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.190668 2375 raft_consensus.cc:385] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.190722 2375 raft_consensus.cc:740] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.191867 2378 tablet_bootstrap.cc:654] T 046a34f9408b406ba08471421f422064 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.196417 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 571) took 7723 ms. Trace:
I20251026 02:56:41.196913 2377 rpcz_store.cc:276] 1026 02:56:33.473193 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.473247 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:41.064914 (+7591667us) ts_tablet_manager.cc:1151] Deleting tablet 634d913c98c842a49d97a54f23752d26
1026 02:56:41.064928 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.075314 (+ 10386us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.144308 (+ 68994us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.196390 (+ 52082us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7591488}
I20251026 02:56:41.198212 2375 consensus_queue.cc:260] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.198341 1945 tablet_service.cc:1505] Processing CreateTablet for tablet be60e04675474c1ea4a69506bc35e68a (DEFAULT_TABLE table=default.Table_27c4161c9b184347971d3035b388e746 [id=03c70cb4e3e341e1a20e480ff85b4034]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.198551 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 572) took 7705 ms. Trace:
I20251026 02:56:41.198622 2377 rpcz_store.cc:276] 1026 02:56:33.493057 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.493134 (+ 77us) service_pool.cc:225] Handling call
1026 02:56:41.198389 (+7705255us) ts_tablet_manager.cc:1151] Deleting tablet 3e38f282e8414b7db2a6d5f4a8933e84
1026 02:56:41.198412 (+ 23us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.198534 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7705033}
I20251026 02:56:41.198756 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be60e04675474c1ea4a69506bc35e68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.198928 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 573) took 7705 ms. Trace:
I20251026 02:56:41.198983 2377 rpcz_store.cc:276] 1026 02:56:33.493570 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.493675 (+ 105us) service_pool.cc:225] Handling call
1026 02:56:41.198839 (+7705164us) ts_tablet_manager.cc:1151] Deleting tablet 892a5fab2af1432eb31d92d9859ea2a0
1026 02:56:41.198848 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.198917 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7705009}
I20251026 02:56:41.199232 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 574) took 7705 ms. Trace:
I20251026 02:56:41.199288 2377 rpcz_store.cc:276] 1026 02:56:33.494008 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.494055 (+ 47us) service_pool.cc:225] Handling call
1026 02:56:41.199134 (+7705079us) ts_tablet_manager.cc:1151] Deleting tablet 34c6a7c774044411a4733b115080c4c6
1026 02:56:41.199141 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.199220 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7704924}
I20251026 02:56:41.199571 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 575) took 7704 ms. Trace:
I20251026 02:56:41.199632 2377 rpcz_store.cc:276] 1026 02:56:33.494634 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.494702 (+ 68us) service_pool.cc:225] Handling call
1026 02:56:41.199466 (+7704764us) ts_tablet_manager.cc:1151] Deleting tablet d0f1d6967931482e9de40b0844da6315
1026 02:56:41.199473 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.199556 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7704630}
I20251026 02:56:41.199839 2377 tablet_replica.cc:333] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:41.200023 2377 raft_consensus.cc:2243] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.200273 2377 raft_consensus.cc:2272] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.200825 2375 ts_tablet_manager.cc:1434] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20251026 02:56:41.200946 2377 ts_tablet_manager.cc:1916] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.201128 2375 tablet_bootstrap.cc:492] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:41.202701 2375 tablet_bootstrap.cc:654] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.204177 2378 tablet_bootstrap.cc:492] T 046a34f9408b406ba08471421f422064 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:41.204548 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 634d913c98c842a49d97a54f23752d26 (table default.Table_d2dcec579c684b92a92d98fb59ee3da1 [id=4f7edfb6d75c4a0b94ca99bb9cfe14f3]) successfully deleted
W20251026 02:56:41.204852 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3e38f282e8414b7db2a6d5f4a8933e84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e38f282e8414b7db2a6d5f4a8933e84 already in progress: creating tablet
I20251026 02:56:41.204998 2378 ts_tablet_manager.cc:1403] T 046a34f9408b406ba08471421f422064 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
W20251026 02:56:41.205122 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 892a5fab2af1432eb31d92d9859ea2a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 892a5fab2af1432eb31d92d9859ea2a0 already in progress: creating tablet
W20251026 02:56:41.205329 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 34c6a7c774044411a4733b115080c4c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34c6a7c774044411a4733b115080c4c6 already in progress: creating tablet
W20251026 02:56:41.205614 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet d0f1d6967931482e9de40b0844da6315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0f1d6967931482e9de40b0844da6315 already in progress: creating tablet
I20251026 02:56:41.205695 2378 raft_consensus.cc:359] T 046a34f9408b406ba08471421f422064 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.215323 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 9d55e3a643d744e6b2d4485543d4b626 (DEFAULT_TABLE table=default.Table_8602d75889ea4576ae575b55fe95e940 [id=b0104213af32435589ae19109b7aee7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.215461 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 662f2f0dfc804ddbbc4b71118a71d048 (DEFAULT_TABLE table=default.Table_f41eab2536d54c839327b1084896bba4 [id=8fb2c3acbda445068bbf1706e7e8e49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.215759 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d55e3a643d744e6b2d4485543d4b626. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.215986 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 662f2f0dfc804ddbbc4b71118a71d048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.221347 2378 raft_consensus.cc:385] T 046a34f9408b406ba08471421f422064 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.221517 2378 raft_consensus.cc:740] T 046a34f9408b406ba08471421f422064 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.204564 2027 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:41.213201 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 662f2f0dfc804ddbbc4b71118a71d048 (DEFAULT_TABLE table=default.Table_f41eab2536d54c839327b1084896bba4 [id=8fb2c3acbda445068bbf1706e7e8e49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.223040 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 662f2f0dfc804ddbbc4b71118a71d048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.223078 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 9d55e3a643d744e6b2d4485543d4b626 (DEFAULT_TABLE table=default.Table_8602d75889ea4576ae575b55fe95e940 [id=b0104213af32435589ae19109b7aee7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.223554 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d55e3a643d744e6b2d4485543d4b626. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.228277 2378 consensus_queue.cc:260] T 046a34f9408b406ba08471421f422064 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.211225 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 9d55e3a643d744e6b2d4485543d4b626 (DEFAULT_TABLE table=default.Table_8602d75889ea4576ae575b55fe95e940 [id=b0104213af32435589ae19109b7aee7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.228917 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d55e3a643d744e6b2d4485543d4b626. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.232580 2233 tablet_service.cc:1505] Processing CreateTablet for tablet be60e04675474c1ea4a69506bc35e68a (DEFAULT_TABLE table=default.Table_27c4161c9b184347971d3035b388e746 [id=03c70cb4e3e341e1a20e480ff85b4034]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.233511 2378 ts_tablet_manager.cc:1434] T 046a34f9408b406ba08471421f422064 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20251026 02:56:41.234408 2378 tablet_bootstrap.cc:492] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:41.235720 2372 ts_tablet_manager.cc:1929] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.235862 2372 log.cc:1199] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/4a5b68772a66402bbb0b7e00bf96fcc0
I20251026 02:56:41.214471 2083 tablet_service.cc:1505] Processing CreateTablet for tablet be60e04675474c1ea4a69506bc35e68a (DEFAULT_TABLE table=default.Table_27c4161c9b184347971d3035b388e746 [id=03c70cb4e3e341e1a20e480ff85b4034]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.236291 2372 ts_tablet_manager.cc:1950] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:41.236423 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be60e04675474c1ea4a69506bc35e68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.236658 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 598) took 7568 ms. Trace:
I20251026 02:56:41.238029 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be60e04675474c1ea4a69506bc35e68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.254894 1656 catalog_manager.cc:5649] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:41.255470 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 8862cd48a5744c99b32f365cb77fa7c4 (table default.Table_dabd227b99964351870151b7731b8175 [id=604570227c014f66aeae89d93edaaafe]). Expected version 1 got 0
W20251026 02:56:41.264187 3496 log.cc:927] Time spent T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
W20251026 02:56:41.266098 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1406c22ee4ee4f94acecad9546208e79 consensus state has no leader
I20251026 02:56:41.266942 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 94bdaa97e7604040b7fc94fe22e743bf
I20251026 02:56:41.267834 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 352a284902c94f088e2e6ae006162db0
I20251026 02:56:41.270077 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 94bdaa97e7604040b7fc94fe22e743bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:41.270293 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 352a284902c94f088e2e6ae006162db0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:41.273746 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 51bfa49903d34ff18fb5a0bb9b75295a (DEFAULT_TABLE table=default.Table_56366720e640479392e1652794ea3847 [id=c13186d1cf86463098c81cfb1dc3b465]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.274204 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51bfa49903d34ff18fb5a0bb9b75295a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.279314 1978 raft_consensus.cc:3060] T 046a34f9408b406ba08471421f422064 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.283073 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 83416598159a4eb598cf91a4a4e6f44a (DEFAULT_TABLE table=default.Table_6c5a2f23ffbf41d294fbf17be375e630 [id=5683c698c2ba46c98852c05330169343]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.283486 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83416598159a4eb598cf91a4a4e6f44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.285485 2378 tablet_bootstrap.cc:654] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.285822 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 94bdaa97e7604040b7fc94fe22e743bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.286056 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 352a284902c94f088e2e6ae006162db0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.287102 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 83416598159a4eb598cf91a4a4e6f44a (DEFAULT_TABLE table=default.Table_6c5a2f23ffbf41d294fbf17be375e630 [id=5683c698c2ba46c98852c05330169343]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.286401 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 2d7858c3245e4a949cbcb1accf8de201 (DEFAULT_TABLE table=default.Table_4a0e936ac9b94d1eb3da682e6bfec5c8 [id=c331e83dc5a04201a57725de16500c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.291639 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d7858c3245e4a949cbcb1accf8de201. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:41.295970 2194 consensus_peers.cc:597] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:41.298666 2377 ts_tablet_manager.cc:1929] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:41.298856 2377 log.cc:1199] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/049a915b35224cdaa38df77c734a0f01
I20251026 02:56:41.299319 2377 ts_tablet_manager.cc:1950] T 049a915b35224cdaa38df77c734a0f01 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:41.269044 3643 raft_consensus.cc:493] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.300293 3643 raft_consensus.cc:515] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.300966 3643 leader_election.cc:290] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.302105 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c1e63ac8c5645f3ba1578792c0a0dae" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:41.303727 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 2d7858c3245e4a949cbcb1accf8de201 (DEFAULT_TABLE table=default.Table_4a0e936ac9b94d1eb3da682e6bfec5c8 [id=c331e83dc5a04201a57725de16500c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.304407 3643 raft_consensus.cc:493] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.304539 3643 raft_consensus.cc:515] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:41.304610 2053 leader_election.cc:343] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.269078 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 94bdaa97e7604040b7fc94fe22e743bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:41.269893 2083 tablet_service.cc:1552] Processing DeleteTablet for tablet 352a284902c94f088e2e6ae006162db0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:41.274138 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 662f2f0dfc804ddbbc4b71118a71d048 (DEFAULT_TABLE table=default.Table_f41eab2536d54c839327b1084896bba4 [id=8fb2c3acbda445068bbf1706e7e8e49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.281585 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 51bfa49903d34ff18fb5a0bb9b75295a (DEFAULT_TABLE table=default.Table_56366720e640479392e1652794ea3847 [id=c13186d1cf86463098c81cfb1dc3b465]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.284950 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 83416598159a4eb598cf91a4a4e6f44a (DEFAULT_TABLE table=default.Table_6c5a2f23ffbf41d294fbf17be375e630 [id=5683c698c2ba46c98852c05330169343]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.305131 3643 leader_election.cc:290] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.305341 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 662f2f0dfc804ddbbc4b71118a71d048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.305541 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51bfa49903d34ff18fb5a0bb9b75295a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.305783 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e483c52e514f1da54335479f59d68e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:41.306277 2053 leader_election.cc:343] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.306777 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c1e63ac8c5645f3ba1578792c0a0dae" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.307792 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e483c52e514f1da54335479f59d68e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.308009 3239 consensus_queue.cc:1048] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:41.308076 3372 raft_consensus.cc:493] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 696476265495492790739ec126a08539)
I20251026 02:56:41.308250 3372 raft_consensus.cc:515] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.308600 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83416598159a4eb598cf91a4a4e6f44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.308848 3372 leader_election.cc:290] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:41.309867 1916 leader_election.cc:343] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 63a63b796191465aab9a63d3ffb0554d
W20251026 02:56:41.310485 2053 leader_election.cc:343] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.310587 2053 leader_election.cc:304] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.310941 3643 raft_consensus.cc:2749] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:41.311605 2053 leader_election.cc:343] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.311697 2053 leader_election.cc:304] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.286756 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 51bfa49903d34ff18fb5a0bb9b75295a (DEFAULT_TABLE table=default.Table_56366720e640479392e1652794ea3847 [id=c13186d1cf86463098c81cfb1dc3b465]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.311851 2372 rpcz_store.cc:276] 1026 02:56:33.668106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.668166 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:41.074531 (+7406365us) ts_tablet_manager.cc:1151] Deleting tablet 4a5b68772a66402bbb0b7e00bf96fcc0
1026 02:56:41.074541 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.081217 (+ 6676us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.235620 (+154403us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.236643 (+ 1023us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7406117}
I20251026 02:56:41.312034 3643 raft_consensus.cc:2749] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:41.312417 3587 raft_consensus.cc:493] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.312530 3587 raft_consensus.cc:515] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.312721 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 4a5b68772a66402bbb0b7e00bf96fcc0 (table default.Table_1e459a892e604d56aeb212131b5a624f [id=8f39b9747e16469eb211e44a69a276e9]) successfully deleted
I20251026 02:56:41.313012 3587 leader_election.cc:290] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.313292 2372 tablet_replica.cc:333] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:41.313472 2372 raft_consensus.cc:2243] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.313624 2372 raft_consensus.cc:2272] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.313620 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db275efdd344817b11a2168675cf95e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:41.314252 2372 ts_tablet_manager.cc:1916] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:41.314257 2053 leader_election.cc:343] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.317687 2378 tablet_bootstrap.cc:492] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:41.318714 2378 ts_tablet_manager.cc:1403] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.000s
I20251026 02:56:41.321817 2378 raft_consensus.cc:359] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.324431 2378 raft_consensus.cc:385] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.324563 2378 raft_consensus.cc:740] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.324791 2378 consensus_queue.cc:260] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.325263 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 2d7858c3245e4a949cbcb1accf8de201 (DEFAULT_TABLE table=default.Table_4a0e936ac9b94d1eb3da682e6bfec5c8 [id=c331e83dc5a04201a57725de16500c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.325752 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d7858c3245e4a949cbcb1accf8de201. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.326265 2378 ts_tablet_manager.cc:1434] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251026 02:56:41.327018 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db275efdd344817b11a2168675cf95e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:41.330086 2053 leader_election.cc:343] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.330238 2053 leader_election.cc:304] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.330659 2378 tablet_bootstrap.cc:492] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:41.330753 3587 raft_consensus.cc:2749] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:41.332846 1915 leader_election.cc:343] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 63a63b796191465aab9a63d3ffb0554d
I20251026 02:56:41.332963 1915 leader_election.cc:304] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:41.333258 2378 tablet_bootstrap.cc:654] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.333413 3372 raft_consensus.cc:2749] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:41.333808 3587 consensus_queue.cc:1048] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:41.344327 2372 ts_tablet_manager.cc:1929] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:41.344545 2372 log.cc:1199] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/358e805370ef410e9d65ee92a8d04f34
I20251026 02:56:41.344967 1982 raft_consensus.cc:3060] T c4b0933f71cb4ac7a58986d16b613e64 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.345628 2372 ts_tablet_manager.cc:1950] T 358e805370ef410e9d65ee92a8d04f34 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:41.345920 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 609) took 7556 ms. Trace:
I20251026 02:56:41.347326 2378 tablet_bootstrap.cc:492] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:41.347447 2378 ts_tablet_manager.cc:1403] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20251026 02:56:41.348091 2378 raft_consensus.cc:359] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.348287 2378 raft_consensus.cc:385] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.348366 2378 raft_consensus.cc:740] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.348662 2378 consensus_queue.cc:260] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.349470 2378 ts_tablet_manager.cc:1434] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:41.349841 2378 tablet_bootstrap.cc:492] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:41.351994 2378 tablet_bootstrap.cc:654] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:41.353507 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7f5cd37c87f34a40bf05829af6de0473 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f5cd37c87f34a40bf05829af6de0473 already in progress: creating tablet
I20251026 02:56:41.345988 2372 rpcz_store.cc:276] 1026 02:56:33.789046 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.789328 (+ 282us) service_pool.cc:225] Handling call
1026 02:56:41.313242 (+7523914us) ts_tablet_manager.cc:1151] Deleting tablet 358e805370ef410e9d65ee92a8d04f34
1026 02:56:41.313252 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.334890 (+ 21638us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.344137 (+ 9247us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.345906 (+ 1769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7523640}
I20251026 02:56:41.354048 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 63eac91f5cb540e6b999ac816ee0b89d (DEFAULT_TABLE table=default.Table_3f4de82346384f0faf6482156a0457d1 [id=2a147f9c71b044b9afd63a5865e34c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.354254 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 358e805370ef410e9d65ee92a8d04f34 (table default.Table_71583d73a49142ae9fec31193fb53542 [id=e86d9fc892a8409590c555e25b27a12f]) successfully deleted
I20251026 02:56:41.354449 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63eac91f5cb540e6b999ac816ee0b89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.354765 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 610) took 7565 ms. Trace:
I20251026 02:56:41.355002 2372 rpcz_store.cc:276] 1026 02:56:33.789270 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.789640 (+ 370us) service_pool.cc:225] Handling call
1026 02:56:41.354616 (+7564976us) ts_tablet_manager.cc:1151] Deleting tablet 9b0406ceb60b48969ca84f6268b43dbb
1026 02:56:41.354630 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.354744 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7564723}
W20251026 02:56:41.355489 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9b0406ceb60b48969ca84f6268b43dbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b0406ceb60b48969ca84f6268b43dbb already in progress: creating tablet
I20251026 02:56:41.355799 2372 tablet_replica.cc:333] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:41.356076 2372 raft_consensus.cc:2243] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.356518 2372 raft_consensus.cc:2272] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.357265 2372 ts_tablet_manager.cc:1916] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.353590 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 63eac91f5cb540e6b999ac816ee0b89d (DEFAULT_TABLE table=default.Table_3f4de82346384f0faf6482156a0457d1 [id=2a147f9c71b044b9afd63a5865e34c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.357707 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63eac91f5cb540e6b999ac816ee0b89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.359767 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83416598159a4eb598cf91a4a4e6f44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.363063 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 886fe7ffa394440eb893887a9feaff2b (DEFAULT_TABLE table=default.Table_46eca4a0a83f4d3091c55183b402758d [id=8f21850f92904d8e92f1b76b2e60fdb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.363543 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886fe7ffa394440eb893887a9feaff2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.353269 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 690) took 3850 ms. Trace:
I20251026 02:56:41.364920 2376 rpcz_store.cc:276] 1026 02:56:37.502543 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.503099 (+ 556us) service_pool.cc:225] Handling call
1026 02:56:41.353141 (+3850042us) ts_tablet_manager.cc:1151] Deleting tablet 8aa0575af07e433c8d96acab31edb4af
1026 02:56:41.353162 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.353253 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3849847}
W20251026 02:56:41.365563 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 8aa0575af07e433c8d96acab31edb4af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aa0575af07e433c8d96acab31edb4af already in progress: creating tablet
I20251026 02:56:41.365934 2376 tablet_replica.cc:333] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:41.366197 2376 raft_consensus.cc:2243] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.367076 2376 raft_consensus.cc:2272] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.367652 2376 ts_tablet_manager.cc:1916] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.367969 3372 raft_consensus.cc:493] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.370705 3372 raft_consensus.cc:515] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.371347 3372 leader_election.cc:290] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:41.371980 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8688a0bf946841deb5d88c931538b129" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:41.372061 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 886fe7ffa394440eb893887a9feaff2b (DEFAULT_TABLE table=default.Table_46eca4a0a83f4d3091c55183b402758d [id=8f21850f92904d8e92f1b76b2e60fdb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.372452 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886fe7ffa394440eb893887a9feaff2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.373210 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 587) took 7741 ms. Trace:
I20251026 02:56:41.373303 2377 rpcz_store.cc:276] 1026 02:56:33.631988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.632060 (+ 72us) service_pool.cc:225] Handling call
1026 02:56:41.199801 (+7567741us) ts_tablet_manager.cc:1151] Deleting tablet 049a915b35224cdaa38df77c734a0f01
1026 02:56:41.199809 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.249245 (+ 49436us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.298548 (+ 49303us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.373195 (+ 74647us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7567517}
I20251026 02:56:41.373598 2377 tablet_replica.cc:333] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:41.373800 2377 raft_consensus.cc:2243] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.373936 2377 raft_consensus.cc:2272] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.374295 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d7858c3245e4a949cbcb1accf8de201. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.376694 2376 ts_tablet_manager.cc:1929] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.380471 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51bfa49903d34ff18fb5a0bb9b75295a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.381603 2377 ts_tablet_manager.cc:1916] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.385789 2381 tablet_bootstrap.cc:700] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539: Time spent opening tablet: real 0.329s user 0.022s sys 0.000s
I20251026 02:56:41.386247 2381 tablet_bootstrap.cc:654] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.386194 1946 tablet_service.cc:1505] Processing CreateTablet for tablet e91466733c4142a7b2879334c0b67aaa (DEFAULT_TABLE table=default.Table_a905e23c621542fa802270f96a8dfd8f [id=e0ec2d9a20a045f4ae8bb5e002ce02e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.388443 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91466733c4142a7b2879334c0b67aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.388993 2378 tablet_bootstrap.cc:492] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:41.389117 2378 ts_tablet_manager.cc:1403] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20251026 02:56:41.389787 2378 raft_consensus.cc:359] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.389947 2378 raft_consensus.cc:385] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.389993 2378 raft_consensus.cc:740] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.390195 2378 consensus_queue.cc:260] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.390827 2378 ts_tablet_manager.cc:1434] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:41.391108 2378 tablet_bootstrap.cc:492] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:41.392727 2378 tablet_bootstrap.cc:654] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.396081 2376 log.cc:1199] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/863ce9e1adb444b7b1399b968ea79bb4
I20251026 02:56:41.396347 2372 ts_tablet_manager.cc:1929] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.396955 2376 ts_tablet_manager.cc:1950] T 863ce9e1adb444b7b1399b968ea79bb4 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:41.397128 2372 log.cc:1199] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3c3050e72d42417ab58a05f7e3aa56b3
I20251026 02:56:41.397563 2372 ts_tablet_manager.cc:1950] T 3c3050e72d42417ab58a05f7e3aa56b3 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:41.397065 1980 raft_consensus.cc:3060] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.398007 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 693) took 3831 ms. Trace:
I20251026 02:56:41.398499 2377 ts_tablet_manager.cc:1929] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:41.398617 2377 log.cc:1199] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/ed836bb7519e4b0290e6547fa57128bc
I20251026 02:56:41.398638 2376 rpcz_store.cc:276] 1026 02:56:37.566963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.569135 (+ 2172us) service_pool.cc:225] Handling call
1026 02:56:41.365884 (+3796749us) ts_tablet_manager.cc:1151] Deleting tablet 863ce9e1adb444b7b1399b968ea79bb4
1026 02:56:41.365894 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.373779 (+ 7885us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.376617 (+ 2838us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.397992 (+ 21375us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3796558}
I20251026 02:56:41.398872 2377 ts_tablet_manager.cc:1950] T ed836bb7519e4b0290e6547fa57128bc P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:41.399133 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 582) took 7800 ms. Trace:
I20251026 02:56:41.399209 2377 rpcz_store.cc:276] 1026 02:56:33.598374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.598435 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:41.373552 (+7775117us) ts_tablet_manager.cc:1151] Deleting tablet ed836bb7519e4b0290e6547fa57128bc
1026 02:56:41.373561 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.385110 (+ 11549us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.398371 (+ 13261us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.399120 (+ 749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7724807}
I20251026 02:56:41.399528 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 863ce9e1adb444b7b1399b968ea79bb4 (table default.Table_03bc36a9fd184c28abd1caf6e5ec4f7a [id=ee13c495b56c48649ceacc8e1122efb4]) successfully deleted
I20251026 02:56:41.400072 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 699) took 3801 ms. Trace:
I20251026 02:56:41.400329 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 611) took 7608 ms. Trace:
I20251026 02:56:41.400502 2376 rpcz_store.cc:276] 1026 02:56:37.598978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.599030 (+ 52us) service_pool.cc:225] Handling call
1026 02:56:41.399953 (+3800923us) ts_tablet_manager.cc:1151] Deleting tablet 6979455632e54d2081065954771f34d3
1026 02:56:41.399962 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.400056 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3776660}
W20251026 02:56:41.401094 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 6979455632e54d2081065954771f34d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6979455632e54d2081065954771f34d3 already in progress: creating tablet
I20251026 02:56:41.401477 2372 rpcz_store.cc:276] 1026 02:56:33.791419 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.791464 (+ 45us) service_pool.cc:225] Handling call
1026 02:56:41.355753 (+7564289us) ts_tablet_manager.cc:1151] Deleting tablet 3c3050e72d42417ab58a05f7e3aa56b3
1026 02:56:41.355763 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.377333 (+ 21570us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.380350 (+ 3017us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.400306 (+ 19956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7564120}
I20251026 02:56:41.402037 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 700) took 3752 ms. Trace:
I20251026 02:56:41.402473 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3c3050e72d42417ab58a05f7e3aa56b3 (table default.Table_1fba8934d98d4fdb8b2c9a0b5d01e810 [id=b9b139445a9040398621353ad052f3c2]) successfully deleted
I20251026 02:56:41.400277 3517 raft_consensus.cc:493] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.402768 2377 tablet_replica.cc:333] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:41.402727 2376 rpcz_store.cc:276] 1026 02:56:37.649266 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.650390 (+ 1124us) service_pool.cc:225] Handling call
1026 02:56:41.401905 (+3751515us) ts_tablet_manager.cc:1151] Deleting tablet d7313a6396c541d7ac6d8ca2e219af90
1026 02:56:41.401918 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.402018 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3751321}
I20251026 02:56:41.403105 2372 tablet_replica.cc:333] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:41.402709 3517 raft_consensus.cc:515] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.403012 1946 tablet_service.cc:1505] Processing CreateTablet for tablet d361c9c955c94cd7bbc59d5705bf7362 (DEFAULT_TABLE table=default.Table_55e6f4711fcc48eda34b1f7b62a7ad2c [id=a9124dacb1bc4c5bb9da53a5f8fbecae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.403481 2377 raft_consensus.cc:2243] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:41.403621 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d361c9c955c94cd7bbc59d5705bf7362. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.403810 2377 raft_consensus.cc:2272] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.400990 3622 consensus_queue.cc:1048] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:41.403811 3517 leader_election.cc:290] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:41.404908 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e701d4023fee45d18c48997aab3227c2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:41.405371 2194 leader_election.cc:343] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:41.406661 2194 consensus_peers.cc:597] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:41.408775 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet d7313a6396c541d7ac6d8ca2e219af90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7313a6396c541d7ac6d8ca2e219af90 already in progress: creating tablet
I20251026 02:56:41.408963 2099 tablet_service.cc:1505] Processing CreateTablet for tablet e91466733c4142a7b2879334c0b67aaa (DEFAULT_TABLE table=default.Table_a905e23c621542fa802270f96a8dfd8f [id=e0ec2d9a20a045f4ae8bb5e002ce02e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.409186 2372 raft_consensus.cc:2243] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:41.409482 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 702) took 3674 ms. Trace:
I20251026 02:56:41.409857 2372 raft_consensus.cc:2272] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.409976 2376 rpcz_store.cc:276] 1026 02:56:37.734705 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.734763 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:41.409370 (+3674607us) ts_tablet_manager.cc:1151] Deleting tablet 0ddeba351c264f67b75b66c55c9734b5
1026 02:56:41.409380 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.409465 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3674413}
I20251026 02:56:41.412560 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 705) took 3595 ms. Trace:
W20251026 02:56:41.412564 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0ddeba351c264f67b75b66c55c9734b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ddeba351c264f67b75b66c55c9734b5 already in progress: creating tablet
I20251026 02:56:41.413178 2372 ts_tablet_manager.cc:1916] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.413292 2376 rpcz_store.cc:276] 1026 02:56:37.816856 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.817046 (+ 190us) service_pool.cc:225] Handling call
1026 02:56:41.412457 (+3595411us) ts_tablet_manager.cc:1151] Deleting tablet 9a5670a9e10c4a9db64881e7e14e839e
1026 02:56:41.412467 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.412541 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3595120}
I20251026 02:56:41.413764 2376 tablet_replica.cc:333] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
W20251026 02:56:41.413733 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 9a5670a9e10c4a9db64881e7e14e839e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a5670a9e10c4a9db64881e7e14e839e already in progress: creating tablet
I20251026 02:56:41.413991 2376 raft_consensus.cc:2243] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.414346 2376 raft_consensus.cc:2272] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.414603 2377 ts_tablet_manager.cc:1916] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.415110 2376 ts_tablet_manager.cc:1916] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.409857 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91466733c4142a7b2879334c0b67aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.410490 2093 tablet_service.cc:1505] Processing CreateTablet for tablet d361c9c955c94cd7bbc59d5705bf7362 (DEFAULT_TABLE table=default.Table_55e6f4711fcc48eda34b1f7b62a7ad2c [id=a9124dacb1bc4c5bb9da53a5f8fbecae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.416724 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d361c9c955c94cd7bbc59d5705bf7362. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.420378 3372 raft_consensus.cc:493] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.420516 3372 raft_consensus.cc:515] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.421123 3372 leader_election.cc:290] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:41.422330 1916 leader_election.cc:343] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: dbf2a1ce78154415b23ab6d0efbc1f52
I20251026 02:56:41.423548 2375 tablet_bootstrap.cc:492] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:41.423722 2375 ts_tablet_manager.cc:1403] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.223s user 0.002s sys 0.002s
I20251026 02:56:41.424625 2375 raft_consensus.cc:359] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.424868 2375 raft_consensus.cc:385] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.425179 2381 tablet_bootstrap.cc:492] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:41.425356 2381 ts_tablet_manager.cc:1403] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.376s user 0.024s sys 0.002s
I20251026 02:56:41.424968 2375 raft_consensus.cc:740] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.426016 2381 raft_consensus.cc:359] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.426239 2381 raft_consensus.cc:385] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.426339 2381 raft_consensus.cc:740] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.426569 2381 consensus_queue.cc:260] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.427114 2381 ts_tablet_manager.cc:1434] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:41.427479 2381 tablet_bootstrap.cc:492] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:41.428889 1629 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 147749us
I20251026 02:56:41.430552 2381 tablet_bootstrap.cc:654] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.428402 2376 ts_tablet_manager.cc:1929] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:41.438948 2376 log.cc:1199] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/8688a0bf946841deb5d88c931538b129
I20251026 02:56:41.439543 2376 ts_tablet_manager.cc:1950] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:41.440346 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 706) took 3616 ms. Trace:
I20251026 02:56:41.441016 2376 rpcz_store.cc:276] 1026 02:56:37.823822 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.827303 (+ 3481us) service_pool.cc:225] Handling call
1026 02:56:41.413720 (+3586417us) ts_tablet_manager.cc:1151] Deleting tablet 8688a0bf946841deb5d88c931538b129
1026 02:56:41.413730 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.421083 (+ 7353us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.428302 (+ 7219us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.440322 (+ 12020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3586232}
I20251026 02:56:41.442986 2376 tablet_replica.cc:333] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:41.444119 2376 raft_consensus.cc:2243] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.444799 2377 ts_tablet_manager.cc:1929] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.444967 2376 raft_consensus.cc:2272] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.445377 2377 log.cc:1199] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/4a5b68772a66402bbb0b7e00bf96fcc0
I20251026 02:56:41.445803 2376 ts_tablet_manager.cc:1916] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.446031 2377 ts_tablet_manager.cc:1950] T 4a5b68772a66402bbb0b7e00bf96fcc0 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:41.447175 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 592) took 7776 ms. Trace:
I20251026 02:56:41.448017 2377 rpcz_store.cc:276] 1026 02:56:33.670379 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.670436 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:41.402717 (+7732281us) ts_tablet_manager.cc:1151] Deleting tablet 4a5b68772a66402bbb0b7e00bf96fcc0
1026 02:56:41.402729 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.425990 (+ 23261us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.441467 (+ 15477us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.447156 (+ 5689us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7732062}
I20251026 02:56:41.448843 2377 tablet_replica.cc:333] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:41.449461 2377 raft_consensus.cc:2243] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:41.436627 2372 ts_tablet_manager.cc:1929] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.442687 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e701d4023fee45d18c48997aab3227c2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:41.449746 2118 raft_consensus.cc:2468] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:41.449787 2375 consensus_queue.cc:260] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.450220 2372 log.cc:1199] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7062c5ff91e14369ab69a2d199585c55
I20251026 02:56:41.450278 2377 raft_consensus.cc:2272] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.450618 2372 ts_tablet_manager.cc:1950] T 7062c5ff91e14369ab69a2d199585c55 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:41.451539 2377 ts_tablet_manager.cc:1916] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.454012 2375 ts_tablet_manager.cc:1434] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.030s user 0.001s sys 0.000s
I20251026 02:56:41.456646 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 049a915b35224cdaa38df77c734a0f01 (table default.Table_1818605e0bc548938337de5d5fc7a781 [id=f1f9d2ad39b34abbbe989bc3fac62b6c]) successfully deleted
I20251026 02:56:41.457386 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet ed836bb7519e4b0290e6547fa57128bc (table default.Table_ec68beb934274468a4f028b3dbaf2cc0 [id=658f4d8e944a4221a722376dc2137896]) successfully deleted
I20251026 02:56:41.458063 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 8688a0bf946841deb5d88c931538b129 (table default.Table_a6e8714ddccb4139adb6c8320e06500b [id=bb34b75c6fcb4ea6b201d05be4982a75]) successfully deleted
I20251026 02:56:41.462460 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 616) took 7648 ms. Trace:
I20251026 02:56:41.462476 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f98c3b707e00420bbd20263069de3dfd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:41.462616 2372 rpcz_store.cc:276] 1026 02:56:33.814370 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.855347 (+ 40977us) service_pool.cc:225] Handling call
1026 02:56:41.403046 (+7547699us) ts_tablet_manager.cc:1151] Deleting tablet 7062c5ff91e14369ab69a2d199585c55
1026 02:56:41.403060 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.423304 (+ 20244us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.436497 (+ 13193us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.462434 (+ 25937us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7547263}
I20251026 02:56:41.463091 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7062c5ff91e14369ab69a2d199585c55 (table default.Table_d48e174e391f42d6aa0b05c322b5b828 [id=91a568329da84558b31575e5127f873f]) successfully deleted
W20251026 02:56:41.463191 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f98c3b707e00420bbd20263069de3dfd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.464567 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4c19009b9b1a44acad306f3b96488546" } new_table_name: "default.Table_df93f777839e490e8edbf5380bf106ee" modify_external_catalogs: true
I20251026 02:56:41.466625 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_98bf1a1deafd476e85a4d04987d9355d" } new_table_name: "default.Table_ab7d62049e1346b689a2419bedeee76c" modify_external_catalogs: true
I20251026 02:56:41.468248 2381 tablet_bootstrap.cc:492] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:41.468513 2381 ts_tablet_manager.cc:1403] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20251026 02:56:41.469692 2381 raft_consensus.cc:359] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.470012 2381 raft_consensus.cc:385] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.470155 2381 raft_consensus.cc:740] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.470862 2381 consensus_queue.cc:260] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.471814 2381 ts_tablet_manager.cc:1434] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:41.472309 2381 tablet_bootstrap.cc:492] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:41.472486 2053 consensus_peers.cc:597] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:41.473031 2375 tablet_bootstrap.cc:492] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:41.474632 2375 tablet_bootstrap.cc:654] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.476475 2372 tablet_replica.cc:333] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:41.476734 2372 raft_consensus.cc:2243] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.476922 2372 raft_consensus.cc:2272] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.477515 2372 ts_tablet_manager.cc:1916] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.484541 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3aed4584b49a4eccac2ba44454866d49
I20251026 02:56:41.485677 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8688a0bf946841deb5d88c931538b129" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.487740 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbf2a1ce78154415b23ab6d0efbc1f52" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.487972 2260 raft_consensus.cc:2393] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 0b6b0a6c4e144982b67980118cddba48 in this term.
I20251026 02:56:41.488039 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 3aed4584b49a4eccac2ba44454866d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:41.488627 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 63eac91f5cb540e6b999ac816ee0b89d (DEFAULT_TABLE table=default.Table_3f4de82346384f0faf6482156a0457d1 [id=2a147f9c71b044b9afd63a5865e34c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.489305 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63eac91f5cb540e6b999ac816ee0b89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.489789 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 3aed4584b49a4eccac2ba44454866d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:41.490701 2381 tablet_bootstrap.cc:654] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.496021 3647 raft_consensus.cc:493] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.496181 3647 raft_consensus.cc:515] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.496801 3647 leader_election.cc:290] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.497532 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7abcdac4a70b48ad9fbba1bc8a60283b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:41.498039 2053 leader_election.cc:343] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.498273 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 886fe7ffa394440eb893887a9feaff2b (DEFAULT_TABLE table=default.Table_46eca4a0a83f4d3091c55183b402758d [id=8f21850f92904d8e92f1b76b2e60fdb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.498724 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886fe7ffa394440eb893887a9feaff2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.505647 2375 tablet_bootstrap.cc:492] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:41.507607 2376 ts_tablet_manager.cc:1929] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.508432 2375 ts_tablet_manager.cc:1403] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.002s
I20251026 02:56:41.509896 2233 tablet_service.cc:1505] Processing CreateTablet for tablet d361c9c955c94cd7bbc59d5705bf7362 (DEFAULT_TABLE table=default.Table_55e6f4711fcc48eda34b1f7b62a7ad2c [id=a9124dacb1bc4c5bb9da53a5f8fbecae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.510362 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d361c9c955c94cd7bbc59d5705bf7362. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:41.513453 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_36597a9424aa42128d73327bd78be6fd" } modify_external_catalogs: true
I20251026 02:56:41.513685 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_36597a9424aa42128d73327bd78be6fd" } modify_external_catalogs: true
I20251026 02:56:41.520015 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 3aed4584b49a4eccac2ba44454866d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.521649 3750 tablet.cc:1722] T 046a34f9408b406ba08471421f422064 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:41.508953 3754 tablet.cc:1722] T c4b0933f71cb4ac7a58986d16b613e64 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:41.525952 2375 raft_consensus.cc:359] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.526239 2375 raft_consensus.cc:385] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.526360 2375 raft_consensus.cc:740] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.526592 2375 consensus_queue.cc:260] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.526808 2376 log.cc:1199] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/1370b9ad917344c093e5da3d0e9b1cf0
I20251026 02:56:41.511799 2372 ts_tablet_manager.cc:1929] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:41.527633 2372 log.cc:1199] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/d212170f056a44dfb15cfcb7f9fbae2f
I20251026 02:56:41.527935 2372 ts_tablet_manager.cc:1950] T d212170f056a44dfb15cfcb7f9fbae2f P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:41.531565 2375 ts_tablet_manager.cc:1434] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251026 02:56:41.531828 2376 ts_tablet_manager.cc:1950] T 1370b9ad917344c093e5da3d0e9b1cf0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:41.533702 1982 raft_consensus.cc:3060] T 8f3ae3a377af46449c3faca339f86053 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.534531 1978 raft_consensus.cc:3060] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.517210 2378 tablet_bootstrap.cc:492] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:41.536933 2378 ts_tablet_manager.cc:1403] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.146s user 0.009s sys 0.009s
I20251026 02:56:41.509372 3756 tablet.cc:1722] T 046a34f9408b406ba08471421f422064 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:41.539758 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 618) took 7614 ms. Trace:
I20251026 02:56:41.539912 2372 rpcz_store.cc:276] 1026 02:56:33.925007 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.925222 (+ 215us) service_pool.cc:225] Handling call
1026 02:56:41.463395 (+7538173us) ts_tablet_manager.cc:1151] Deleting tablet d212170f056a44dfb15cfcb7f9fbae2f
1026 02:56:41.476431 (+ 13036us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.486970 (+ 10539us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.511031 (+ 24061us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.539735 (+ 28704us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7537989}
I20251026 02:56:41.540459 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 619) took 7614 ms. Trace:
I20251026 02:56:41.540584 2372 rpcz_store.cc:276] 1026 02:56:33.925523 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.925678 (+ 155us) service_pool.cc:225] Handling call
1026 02:56:41.540325 (+7614647us) ts_tablet_manager.cc:1151] Deleting tablet 82e4bc6195484074b0da610a8d1b65bc
1026 02:56:41.540335 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.540443 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7614509}
I20251026 02:56:41.540972 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 634) took 6934 ms. Trace:
I20251026 02:56:41.541219 2372 rpcz_store.cc:276] 1026 02:56:34.606481 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.606527 (+ 46us) service_pool.cc:225] Handling call
1026 02:56:41.540877 (+6934350us) ts_tablet_manager.cc:1151] Deleting tablet 6699b50d87964f6a89c365171253ad25
1026 02:56:41.540885 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.540958 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6888563}
I20251026 02:56:41.541568 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 635) took 6934 ms. Trace:
I20251026 02:56:41.541682 2372 rpcz_store.cc:276] 1026 02:56:34.606734 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.606775 (+ 41us) service_pool.cc:225] Handling call
1026 02:56:41.541481 (+6934706us) ts_tablet_manager.cc:1151] Deleting tablet 5824831b8e184daf8b40543686c06467
1026 02:56:41.541489 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.541555 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6888874}
I20251026 02:56:41.542037 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 636) took 6935 ms. Trace:
I20251026 02:56:41.542165 2372 rpcz_store.cc:276] 1026 02:56:34.606972 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.607010 (+ 38us) service_pool.cc:225] Handling call
1026 02:56:41.541934 (+6934924us) ts_tablet_manager.cc:1151] Deleting tablet 848e414f6e134761818039f7e98500ac
1026 02:56:41.541942 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.542024 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6889267}
I20251026 02:56:41.542483 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 639) took 6934 ms. Trace:
I20251026 02:56:41.542593 2372 rpcz_store.cc:276] 1026 02:56:34.607730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.608139 (+ 409us) service_pool.cc:225] Handling call
1026 02:56:41.542394 (+6934255us) ts_tablet_manager.cc:1151] Deleting tablet 5ddc7972f29a429ea51f6c6c1a074baf
1026 02:56:41.542402 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.542471 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6886032}
I20251026 02:56:41.542930 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 650) took 6587 ms. Trace:
I20251026 02:56:41.543027 2372 rpcz_store.cc:276] 1026 02:56:34.954965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.955006 (+ 41us) service_pool.cc:225] Handling call
1026 02:56:41.542849 (+6587843us) ts_tablet_manager.cc:1151] Deleting tablet c09153fbfe18434393c188058a3461d5
1026 02:56:41.542857 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.542917 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6553570}
I20251026 02:56:41.543406 2372 tablet_replica.cc:333] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:41.543623 2372 raft_consensus.cc:2243] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:41.539444 3753 tablet.cc:1722] T c4b0933f71cb4ac7a58986d16b613e64 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:41.544617 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 708) took 3684 ms. Trace:
I20251026 02:56:41.544816 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet d212170f056a44dfb15cfcb7f9fbae2f (table default.Table_7667f4f99bf744579f1dc2b2d0f469eb [id=822eea61b89846f4828511c58a9da4c8]) successfully deleted
W20251026 02:56:41.545130 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 82e4bc6195484074b0da610a8d1b65bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82e4bc6195484074b0da610a8d1b65bc already in progress: creating tablet
I20251026 02:56:41.545588 2381 tablet_bootstrap.cc:492] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
W20251026 02:56:41.545586 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 6699b50d87964f6a89c365171253ad25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6699b50d87964f6a89c365171253ad25 already in progress: creating tablet
W20251026 02:56:41.545820 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 5824831b8e184daf8b40543686c06467 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5824831b8e184daf8b40543686c06467 already in progress: creating tablet
I20251026 02:56:41.545964 2381 ts_tablet_manager.cc:1403] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.074s user 0.005s sys 0.008s
W20251026 02:56:41.546110 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 848e414f6e134761818039f7e98500ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 848e414f6e134761818039f7e98500ac already in progress: creating tablet
W20251026 02:56:41.546311 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 5ddc7972f29a429ea51f6c6c1a074baf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ddc7972f29a429ea51f6c6c1a074baf already in progress: creating tablet
W20251026 02:56:41.546695 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet c09153fbfe18434393c188058a3461d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c09153fbfe18434393c188058a3461d5 already in progress: creating tablet
W20251026 02:56:41.547581 1649 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet b8fe7825fa8d46669772cf84bc4b3afc (table default.Table_c51b98a0c9e54daf9d1e457f34d56475 [id=649f153da3b14f0b99acef230e010a8c], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet b8fe7825fa8d46669772cf84bc4b3afc consensus state has no leader
I20251026 02:56:41.546515 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 4a5b68772a66402bbb0b7e00bf96fcc0 (table default.Table_1e459a892e604d56aeb212131b5a624f [id=8f39b9747e16469eb211e44a69a276e9]) successfully deleted
I20251026 02:56:41.547691 2381 raft_consensus.cc:359] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.548040 2381 raft_consensus.cc:385] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.548189 2381 raft_consensus.cc:740] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.548595 2381 consensus_queue.cc:260] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.549593 2377 ts_tablet_manager.cc:1929] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.549952 2377 log.cc:1199] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/5caeb11322fe48348ebb523967152187
I20251026 02:56:41.550324 2377 ts_tablet_manager.cc:1950] T 5caeb11322fe48348ebb523967152187 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:41.550768 2381 ts_tablet_manager.cc:1434] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:41.566448 2376 rpcz_store.cc:276] 1026 02:56:37.860193 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.868285 (+ 8092us) service_pool.cc:225] Handling call
1026 02:56:41.442939 (+3574654us) ts_tablet_manager.cc:1151] Deleting tablet 1370b9ad917344c093e5da3d0e9b1cf0
1026 02:56:41.442948 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.499292 (+ 56344us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.507505 (+ 8213us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.544578 (+ 37073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3574442}
I20251026 02:56:41.567273 2376 tablet_replica.cc:333] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:41.567567 2376 raft_consensus.cc:2243] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.567793 2376 raft_consensus.cc:2272] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.568466 1980 raft_consensus.cc:3060] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.568888 2376 ts_tablet_manager.cc:1916] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.539528 2378 raft_consensus.cc:359] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.574822 2378 raft_consensus.cc:385] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.575193 2378 raft_consensus.cc:740] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.576519 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet e701d4023fee45d18c48997aab3227c2 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:41.576421 2378 consensus_queue.cc:260] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.580343 2378 ts_tablet_manager.cc:1434] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
W20251026 02:56:41.584777 2194 consensus_peers.cc:597] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1370b9ad917344c093e5da3d0e9b1cf0. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:41.586820 2372 raft_consensus.cc:2272] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.586896 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 1370b9ad917344c093e5da3d0e9b1cf0 (table default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]) successfully deleted
I20251026 02:56:41.588786 3372 raft_consensus.cc:493] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 696476265495492790739ec126a08539)
I20251026 02:56:41.588953 3372 raft_consensus.cc:515] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.589686 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe (table default.Table_47994972d3e0469c9acc96120aad4e1f [id=87349d9c6c764382b828e716e7854c59]). Expected version 3 got 0
I20251026 02:56:41.543978 2375 tablet_bootstrap.cc:492] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:41.590021 3372 leader_election.cc:290] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:41.588492 3643 raft_consensus.cc:493] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.590148 3643 raft_consensus.cc:515] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.590776 3643 leader_election.cc:290] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.591431 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2b3b66c0fb3a4f10bf684d40855009d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:41.592092 2375 tablet_bootstrap.cc:654] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.592785 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8dc3bc43f2194fba9f556ad3a4530cc3" } new_table_name: "default.Table_bd556a71941d43b188d2ab5a25c25083" modify_external_catalogs: true
I20251026 02:56:41.593966 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet ce1ab21ddf8e45bc9063b63db9c1b97f (table default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5]). Expected version 1 got 0
I20251026 02:56:41.594087 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "386e1bd4635b4a7e8e7f249b577fd978" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:41.594451 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet e9cd10092aac40068a1c7f85e514fd37 (table default.Table_fe09be2527a840b09e3e64c27965845d [id=de5272bcc29644208ddd95bdec21254e]). Expected version 1 got 0
W20251026 02:56:41.594626 2053 leader_election.cc:343] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.597491 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_dab46018e72e4863ba644a72c0fcdbf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.598423 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e326907a7b444969c82df0e066be160 has no consensus state
I20251026 02:56:41.598289 1915 leader_election.cc:304] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:41.598438 2107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d65b0d79e2e54af9810022b683c3288b" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:41.598583 1915 raft_consensus.cc:2686] T 8688a0bf946841deb5d88c931538b129 P 7fef01f4f2b843538ea245a09dd087a8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251026 02:56:41.582887 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5b06ba4234b24bf59077dcc622039de2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:41.597728 2378 tablet_bootstrap.cc:492] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:41.599614 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b06ba4234b24bf59077dcc622039de2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.600611 1915 leader_election.cc:304] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:41.600986 3372 raft_consensus.cc:3060] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:41.602295 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b3b66c0fb3a4f10bf684d40855009d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.602736 2372 ts_tablet_manager.cc:1916] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.603819 3372 raft_consensus.cc:2749] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:41.606295 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dab46018e72e4863ba644a72c0fcdbf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.607613 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 046a34f9408b406ba08471421f422064 (table default.Table_760b799491d7426a86aa1dc3c76fab31 [id=bc8a2d95ed134fae9f3e9deeac4f130a]). Expected version 3 got 0
I20251026 02:56:41.607523 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_072c979a175d433b8ee402eeba321771"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.608412 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_072c979a175d433b8ee402eeba321771 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.609758 1657 catalog_manager.cc:5649] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:41.610209 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet ff242ef9541a42d793c4484190956da6 (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]). Expected version 1 got 0
I20251026 02:56:41.610738 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 5e9b0e1d3f3f4fdbb8c58f26a4308771 (table default.Table_1fa437ef7f89450bb6f672a1fb491baa [id=44c37945ff9f48e8b2c07648394ac177]). Expected version 1 got 0
I20251026 02:56:41.612010 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet dba102838945453cbf44e1cfe505a52c
I20251026 02:56:41.612854 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet dba102838945453cbf44e1cfe505a52c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:41.610047 2242 tablet_service.cc:1505] Processing CreateTablet for tablet e91466733c4142a7b2879334c0b67aaa (DEFAULT_TABLE table=default.Table_a905e23c621542fa802270f96a8dfd8f [id=e0ec2d9a20a045f4ae8bb5e002ce02e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:41.613442 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_79f94fa4977d4066907d585cd301d523" } new_table_name: "default.Table_f3f28520bd3a44738d0e86f989470e12" modify_external_catalogs: true
I20251026 02:56:41.616281 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet dba102838945453cbf44e1cfe505a52c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:41.618979 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5aff38ba32864ad18b5d808e5235efe3 consensus state has no leader
I20251026 02:56:41.619319 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:41.619956 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_bff4472649e4461a8f829b44bb15da12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.620573 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bff4472649e4461a8f829b44bb15da12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.620842 2381 tablet_bootstrap.cc:492] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:41.621236 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 600) took 7846 ms. Trace:
I20251026 02:56:41.621320 2377 rpcz_store.cc:276] 1026 02:56:33.775219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.775379 (+ 160us) service_pool.cc:225] Handling call
1026 02:56:41.448795 (+7673416us) ts_tablet_manager.cc:1151] Deleting tablet 5caeb11322fe48348ebb523967152187
1026 02:56:41.448806 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.459997 (+ 11191us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.549490 (+ 89493us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.621214 (+ 71724us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7670054}
I20251026 02:56:41.621692 2377 tablet_replica.cc:333] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:41.621857 2377 raft_consensus.cc:2243] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:41.622185 2377 raft_consensus.cc:2272] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.622687 2381 tablet_bootstrap.cc:654] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.622769 2377 ts_tablet_manager.cc:1916] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.629169 2377 ts_tablet_manager.cc:1929] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.629307 2377 log.cc:1199] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/0fe0e06a86e241bc91a1db4219fe25bc
I20251026 02:56:41.629681 2377 ts_tablet_manager.cc:1950] T 0fe0e06a86e241bc91a1db4219fe25bc P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:41.630445 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 599) took 7855 ms. Trace:
I20251026 02:56:41.630534 2377 rpcz_store.cc:276] 1026 02:56:33.775143 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.775549 (+ 406us) service_pool.cc:225] Handling call
1026 02:56:41.621651 (+7846102us) ts_tablet_manager.cc:1151] Deleting tablet 0fe0e06a86e241bc91a1db4219fe25bc
1026 02:56:41.621660 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.625959 (+ 4299us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.629067 (+ 3108us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.630422 (+ 1355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7841556}
I20251026 02:56:41.630843 2377 tablet_replica.cc:333] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:41.631044 2377 raft_consensus.cc:2243] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:41.631399 2377 raft_consensus.cc:2272] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.631815 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 887dc91626664f23b39cf70a6bb9381e
W20251026 02:56:41.632543 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0dea6ffb68834373a03ced4f0f804473 consensus state has no leader
I20251026 02:56:41.632984 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_74606f0d7edf4d7281d9d30c1e6f052f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:41.633589 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91466733c4142a7b2879334c0b67aaa. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:41.633720 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74606f0d7edf4d7281d9d30c1e6f052f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.634047 3643 raft_consensus.cc:493] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.634176 3643 raft_consensus.cc:515] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.634845 3643 leader_election.cc:290] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.635337 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fb320a7039df4723a13ea9706037a238"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:41.635628 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7abcdac4a70b48ad9fbba1bc8a60283b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.635682 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dbe8002df6c4b64b8ec8260ca6b9cf5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:41.635843 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb320a7039df4723a13ea9706037a238 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:41.636281 2053 leader_election.cc:343] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.637307 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7262f683830a4f35ae728338fc3344b8" } new_table_name: "default.Table_4ee67977fc2a49f584e706a98880f542" modify_external_catalogs: true
I20251026 02:56:41.638830 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ecf77fe2da2f4a9eb37ccdb284cd45af" } modify_external_catalogs: true
I20251026 02:56:41.639026 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ecf77fe2da2f4a9eb37ccdb284cd45af" } modify_external_catalogs: true
I20251026 02:56:41.640667 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b67009abd92345148b7cb2d1be01e4a9" } modify_external_catalogs: true
I20251026 02:56:41.640841 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b67009abd92345148b7cb2d1be01e4a9" } modify_external_catalogs: true
I20251026 02:56:41.642438 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 887dc91626664f23b39cf70a6bb9381e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:41.644714 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 887dc91626664f23b39cf70a6bb9381e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:41.645984 2381 tablet_bootstrap.cc:492] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:41.646322 2381 ts_tablet_manager.cc:1403] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.002s
I20251026 02:56:41.645221 2377 ts_tablet_manager.cc:1916] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.654763 3647 raft_consensus.cc:493] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.654970 3647 raft_consensus.cc:515] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.656060 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882cd1680aa140f6a63a961cd4aec25a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:41.656175 3647 leader_election.cc:290] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:41.656623 2053 leader_election.cc:343] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.661820 1656 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet f8dfc861cfa147db98f78ae4e4e75b60
W20251026 02:56:41.662221 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4d64902618d46428ff345bc190b03bd consensus state has no leader
I20251026 02:56:41.662184 3647 raft_consensus.cc:493] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.662312 3647 raft_consensus.cc:515] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:41.662585 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53264ef08ff7454faa19b18fa35bd803 consensus state has no leader
I20251026 02:56:41.663244 3647 leader_election.cc:290] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.664094 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ade0f7de597049ffb45274d60c9c0932
I20251026 02:56:41.666649 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet ade0f7de597049ffb45274d60c9c0932 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:41.647814 2381 raft_consensus.cc:359] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.668382 2381 raft_consensus.cc:385] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.668473 2381 raft_consensus.cc:740] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.668715 2381 consensus_queue.cc:260] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.669314 2381 ts_tablet_manager.cc:1434] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20251026 02:56:41.657635 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "386e1bd4635b4a7e8e7f249b577fd978" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.657815 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dbe8002df6c4b64b8ec8260ca6b9cf5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.670205 2260 raft_consensus.cc:2468] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:41.658205 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882cd1680aa140f6a63a961cd4aec25a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.658766 3537 consensus_queue.cc:1048] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:41.659104 3517 consensus_queue.cc:1048] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:41.674809 2378 tablet_bootstrap.cc:654] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.676577 3754 tablet.cc:1722] T 046a34f9408b406ba08471421f422064 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:41.681010 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8799a5af2b42cc92ccc0195cff98e7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:41.681133 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet ade0f7de597049ffb45274d60c9c0932 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:41.681543 2053 leader_election.cc:343] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.683696 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe (table default.Table_47994972d3e0469c9acc96120aad4e1f [id=87349d9c6c764382b828e716e7854c59]). Expected version 3 got 0
I20251026 02:56:41.683808 3647 raft_consensus.cc:493] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.683947 3647 raft_consensus.cc:515] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.684945 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1ca5b8379c94deab1130bfe8bdc87b0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:41.685076 3647 leader_election.cc:290] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:41.685433 2053 leader_election.cc:343] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.685585 2381 tablet_bootstrap.cc:492] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:41.659375 2195 leader_election.cc:304] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.687408 2381 tablet_bootstrap.cc:654] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.689728 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_07a34faac32549c99cb5b5a936a56945" } modify_external_catalogs: true
I20251026 02:56:41.689962 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_07a34faac32549c99cb5b5a936a56945" } modify_external_catalogs: true
I20251026 02:56:41.691882 3517 raft_consensus.cc:2804] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:41.692000 3517 raft_consensus.cc:493] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:41.692093 3517 raft_consensus.cc:3060] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:41.692200 2195 consensus_peers.cc:597] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c3050e72d42417ab58a05f7e3aa56b3. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:41.695549 3517 raft_consensus.cc:515] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.696983 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e701d4023fee45d18c48997aab3227c2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:41.698122 2194 leader_election.cc:343] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.698642 3753 tablet.cc:1722] T 8f3ae3a377af46449c3faca339f86053 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:41.698899 3517 leader_election.cc:290] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:41.706431 1661 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 8f3ae3a377af46449c3faca339f86053 (table default.Table_3af0e3e691254fbb8f124697619830bf [id=31f977e55ba041f59bcc721d469746fd]). Expected version 3 got 0
I20251026 02:56:41.712158 2308 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:41.713084 3750 tablet.cc:1722] T c4b0933f71cb4ac7a58986d16b613e64 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:41.716967 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ecbed9dd56e94cad8d7387155b934c50
I20251026 02:56:41.717868 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet ecbed9dd56e94cad8d7387155b934c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:41.719363 1663 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_760b799491d7426a86aa1dc3c76fab31 [id=bc8a2d95ed134fae9f3e9deeac4f130a] alter complete (version 3)
I20251026 02:56:41.720717 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet ecbed9dd56e94cad8d7387155b934c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:41.721554 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 5caeb11322fe48348ebb523967152187 (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
I20251026 02:56:41.722941 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 0fe0e06a86e241bc91a1db4219fe25bc (table default.Table_caee34582bea4cd1970c4cc59078f9d1 [id=0494cd534eb24adc94c907471cdb346a]) successfully deleted
W20251026 02:56:41.723702 2053 leader_election.cc:343] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.723814 2053 leader_election.cc:304] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.723992 3647 raft_consensus.cc:2749] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:41.726040 2165 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:41.727355 2053 leader_election.cc:304] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.727813 3647 raft_consensus.cc:2804] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:41.727921 3647 raft_consensus.cc:493] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:41.728024 3647 raft_consensus.cc:3060] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.728628 2375 tablet_bootstrap.cc:492] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:41.728765 2375 ts_tablet_manager.cc:1403] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.185s user 0.005s sys 0.001s
I20251026 02:56:41.729434 2375 raft_consensus.cc:359] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.729595 2375 raft_consensus.cc:385] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.729656 2375 raft_consensus.cc:740] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
W20251026 02:56:41.729499 2053 leader_election.cc:343] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.729769 2053 leader_election.cc:304] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.729854 2375 consensus_queue.cc:260] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.730069 3643 raft_consensus.cc:2749] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:41.730275 2375 ts_tablet_manager.cc:1434] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 02:56:41.730432 3616 raft_consensus.cc:493] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.730590 2375 tablet_bootstrap.cc:492] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:41.730549 3616 raft_consensus.cc:515] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.731074 3616 leader_election.cc:290] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:41.731745 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:41.732098 2375 tablet_bootstrap.cc:654] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.732060 1982 raft_consensus.cc:2468] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:41.732630 2194 leader_election.cc:304] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:41.732945 3616 raft_consensus.cc:2804] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 02:56:41.733014 3616 raft_consensus.cc:493] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:41.733096 3616 raft_consensus.cc:3060] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:41.733413 3647 raft_consensus.cc:515] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.734256 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dbe8002df6c4b64b8ec8260ca6b9cf5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:41.734760 2053 leader_election.cc:343] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.735471 3647 leader_election.cc:290] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.735631 3616 raft_consensus.cc:515] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.736109 3616 leader_election.cc:290] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:41.738194 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47037db3602246a2ab6224557286162f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:41.738359 1978 raft_consensus.cc:3060] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:41.739121 3756 tablet.cc:1722] T 8f3ae3a377af46449c3faca339f86053 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251026 02:56:41.739619 2053 leader_election.cc:343] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.739718 2053 leader_election.cc:304] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.740113 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet ecbed9dd56e94cad8d7387155b934c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.740569 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet ade0f7de597049ffb45274d60c9c0932 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.740773 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.740902 1978 raft_consensus.cc:2468] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 2.
I20251026 02:56:41.741182 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 7177931353164b2aa82c00b2807ccd63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.741629 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e701d4023fee45d18c48997aab3227c2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:41.741776 2120 raft_consensus.cc:3060] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.741942 2381 tablet_bootstrap.cc:492] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:41.742075 2381 ts_tablet_manager.cc:1403] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.008s
I20251026 02:56:41.742817 2381 raft_consensus.cc:359] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.742965 2381 raft_consensus.cc:385] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.743010 2381 raft_consensus.cc:740] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.743185 2381 consensus_queue.cc:260] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.743589 2381 ts_tablet_manager.cc:1434] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 02:56:41.743837 2381 tablet_bootstrap.cc:492] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:41.744679 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1ca5b8379c94deab1130bfe8bdc87b0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.744768 2194 leader_election.cc:304] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:41.744828 2260 raft_consensus.cc:2468] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:41.745306 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dbe8002df6c4b64b8ec8260ca6b9cf5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:41.745466 2256 raft_consensus.cc:3060] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.745584 3616 raft_consensus.cc:2804] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 02:56:41.745783 2381 tablet_bootstrap.cc:654] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.745886 3616 raft_consensus.cc:697] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 2 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:41.746539 3616 consensus_queue.cc:237] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.748086 2256 raft_consensus.cc:2468] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:41.748497 3587 raft_consensus.cc:2749] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:41.749644 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d65b0d79e2e54af9810022b683c3288b" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.749737 3587 raft_consensus.cc:493] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.749838 3587 raft_consensus.cc:515] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.749930 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet ed836bb7519e4b0290e6547fa57128bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.750393 3587 leader_election.cc:290] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.750510 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8799a5af2b42cc92ccc0195cff98e7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.750645 2259 raft_consensus.cc:2468] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:41.751071 2120 raft_consensus.cc:2468] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:41.751405 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 887dc91626664f23b39cf70a6bb9381e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.751636 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c7efb70c0804c149f36440a4a9e6ef8" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:41.752292 2053 leader_election.cc:343] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.757112 2378 tablet_bootstrap.cc:492] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:41.757262 2378 ts_tablet_manager.cc:1403] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.160s user 0.003s sys 0.003s
I20251026 02:56:41.758072 2378 raft_consensus.cc:359] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.758271 2378 raft_consensus.cc:385] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.758329 2378 raft_consensus.cc:740] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.758590 2378 consensus_queue.cc:260] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.759080 2378 ts_tablet_manager.cc:1434] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:41.762933 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b703a0b78fda4c0597aded8f5e8ccf6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.763563 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b703a0b78fda4c0597aded8f5e8ccf6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.763890 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_925beaee6cd548618705f64874644a54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:41.766547 3647 raft_consensus.cc:493] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.766742 3647 raft_consensus.cc:515] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.767939 3647 leader_election.cc:290] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.768256 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa10cd990a944eab8d9eb66aaf20080c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:41.768772 2053 leader_election.cc:343] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:41.769810 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_925beaee6cd548618705f64874644a54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.775926 2378 tablet_bootstrap.cc:492] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:41.778090 2378 tablet_bootstrap.cc:654] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.781533 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 663076e362c44837beb322aa965cde42 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:41.786789 2376 ts_tablet_manager.cc:1929] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.787032 2376 log.cc:1199] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/63a63b796191465aab9a63d3ffb0554d
I20251026 02:56:41.787515 2376 ts_tablet_manager.cc:1950] T 63a63b796191465aab9a63d3ffb0554d P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:41.787746 2377 ts_tablet_manager.cc:1929] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.789167 2377 log.cc:1199] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/2524ee0c964345459c3e62f1e39b8f9e
I20251026 02:56:41.790082 2377 ts_tablet_manager.cc:1950] T 2524ee0c964345459c3e62f1e39b8f9e P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:41.790546 2372 ts_tablet_manager.cc:1929] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.794981 2381 tablet_bootstrap.cc:492] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:41.795266 2381 ts_tablet_manager.cc:1403] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.001s
I20251026 02:56:41.796548 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_64502b8f2393429f85256d6cbdaf7148"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.797211 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64502b8f2393429f85256d6cbdaf7148 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.789601 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet dba102838945453cbf44e1cfe505a52c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.800323 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_890ffb18fe5a49f1938a0fca64189907" } new_table_name: "default.Table_73be634ceddf488ea518d690587f6cd0" modify_external_catalogs: true
W20251026 02:56:41.800518 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 59237.9us
I20251026 02:56:41.802960 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 601) took 8024 ms. Trace:
I20251026 02:56:41.803063 2377 rpcz_store.cc:276] 1026 02:56:33.778660 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.779762 (+ 1102us) service_pool.cc:225] Handling call
1026 02:56:41.630796 (+7851034us) ts_tablet_manager.cc:1151] Deleting tablet 2524ee0c964345459c3e62f1e39b8f9e
1026 02:56:41.630806 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.697513 (+ 66707us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.732094 (+ 34581us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.802940 (+ 70846us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7832042}
I20251026 02:56:41.803392 2377 tablet_replica.cc:333] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:41.803576 2377 raft_consensus.cc:2243] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:41.803900 2377 raft_consensus.cc:2272] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.804304 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 709) took 3922 ms. Trace:
I20251026 02:56:41.804474 2376 rpcz_store.cc:276] 1026 02:56:37.882182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.882259 (+ 77us) service_pool.cc:225] Handling call
1026 02:56:41.567206 (+3684947us) ts_tablet_manager.cc:1151] Deleting tablet 63a63b796191465aab9a63d3ffb0554d
1026 02:56:41.567222 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.572135 (+ 4913us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.786674 (+214539us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.804281 (+ 17607us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3683929}
I20251026 02:56:41.804849 1915 leader_election.cc:304] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:41.805317 3372 raft_consensus.cc:2749] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:41.805861 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 63a63b796191465aab9a63d3ffb0554d (table default.Table_b389bf6a8e5d4ca39fb2c59f8103652c [id=d27916c5ae83463e868d14a5d22ae281]) successfully deleted
I20251026 02:56:41.805038 2381 raft_consensus.cc:359] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.806325 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 710) took 3923 ms. Trace:
I20251026 02:56:41.806416 2376 rpcz_store.cc:276] 1026 02:56:37.882507 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.883474 (+ 967us) service_pool.cc:225] Handling call
1026 02:56:41.806199 (+3922725us) ts_tablet_manager.cc:1151] Deleting tablet b1b6613742a24c21a1db6ebe858a846f
1026 02:56:41.806209 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.806309 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3922565}
W20251026 02:56:41.806921 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b1b6613742a24c21a1db6ebe858a846f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1b6613742a24c21a1db6ebe858a846f already in progress: creating tablet
I20251026 02:56:41.807266 2376 tablet_replica.cc:333] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:41.806659 2381 raft_consensus.cc:385] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.807549 2376 raft_consensus.cc:2243] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.807745 2376 raft_consensus.cc:2272] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.807334 2381 raft_consensus.cc:740] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.808331 2376 ts_tablet_manager.cc:1916] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.809522 2381 consensus_queue.cc:260] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:41.816498 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68ad3d4a2d484badb4b35884517b72f1 consensus state has no leader
I20251026 02:56:41.813473 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet b1ca5b8379c94deab1130bfe8bdc87b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:41.818666 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c84ad81d49374f2aab22ef0c241af795" } modify_external_catalogs: true
I20251026 02:56:41.818835 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c84ad81d49374f2aab22ef0c241af795" } modify_external_catalogs: true
I20251026 02:56:41.819012 2377 ts_tablet_manager.cc:1916] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.813621 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa10cd990a944eab8d9eb66aaf20080c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.813679 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c7efb70c0804c149f36440a4a9e6ef8" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.821326 2378 tablet_bootstrap.cc:492] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:41.821481 2378 ts_tablet_manager.cc:1403] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20251026 02:56:41.821060 2381 ts_tablet_manager.cc:1434] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20251026 02:56:41.822257 2378 raft_consensus.cc:359] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.822444 2378 raft_consensus.cc:385] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.822505 2378 raft_consensus.cc:740] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.822729 2378 consensus_queue.cc:260] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.823848 2381 tablet_bootstrap.cc:492] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:41.814221 2195 leader_election.cc:343] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 47037db3602246a2ab6224557286162f
W20251026 02:56:41.824875 2195 leader_election.cc:343] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 47037db3602246a2ab6224557286162f
I20251026 02:56:41.825165 2195 leader_election.cc:304] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.825205 2378 ts_tablet_manager.cc:1434] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251026 02:56:41.825435 3622 raft_consensus.cc:2804] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:41.825510 2378 tablet_bootstrap.cc:492] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:41.825542 3622 raft_consensus.cc:697] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:41.825750 3622 consensus_queue.cc:237] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.826231 2377 ts_tablet_manager.cc:1929] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:41.826323 2377 log.cc:1199] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/358e805370ef410e9d65ee92a8d04f34
I20251026 02:56:41.826670 2377 ts_tablet_manager.cc:1950] T 358e805370ef410e9d65ee92a8d04f34 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:41.803828 2375 tablet_bootstrap.cc:492] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:41.830018 2375 ts_tablet_manager.cc:1403] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.000s
I20251026 02:56:41.830298 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 2524ee0c964345459c3e62f1e39b8f9e (table default.Table_164cb1784f4645e499339f7163fdfc2b [id=91b4b1a8162441299c680e7c202a96d9]) successfully deleted
I20251026 02:56:41.804448 2053 leader_election.cc:304] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.830889 2375 raft_consensus.cc:359] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.831081 2375 raft_consensus.cc:385] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.831152 2375 raft_consensus.cc:740] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.831205 2053 leader_election.cc:304] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.831410 2375 consensus_queue.cc:260] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.831619 2053 leader_election.cc:304] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.831830 2372 log.cc:1199] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/ae95654d129441e08fcb3f3b355ffc22
I20251026 02:56:41.831365 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_47994972d3e0469c9acc96120aad4e1f" } new_table_name: "default.Table_7244d7be0e764416aaba258478bdffdc" modify_external_catalogs: true
I20251026 02:56:41.833788 3647 raft_consensus.cc:2804] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:41.833868 3647 raft_consensus.cc:493] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:41.833932 3647 raft_consensus.cc:3060] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.836329 2372 ts_tablet_manager.cc:1950] T ae95654d129441e08fcb3f3b355ffc22 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:41.836840 3587 raft_consensus.cc:2804] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:41.836920 3587 raft_consensus.cc:493] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:41.836985 3587 raft_consensus.cc:3060] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.837348 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 660) took 6879 ms. Trace:
I20251026 02:56:41.837459 2372 rpcz_store.cc:276] 1026 02:56:34.958103 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.989308 (+ 31205us) service_pool.cc:225] Handling call
1026 02:56:41.543367 (+6554059us) ts_tablet_manager.cc:1151] Deleting tablet ae95654d129441e08fcb3f3b355ffc22
1026 02:56:41.543375 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.606006 (+ 62631us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.790433 (+184427us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.837325 (+ 46892us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6553893}
I20251026 02:56:41.837998 3643 raft_consensus.cc:2804] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:41.838002 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet ae95654d129441e08fcb3f3b355ffc22 (table default.Table_8602d75889ea4576ae575b55fe95e940 [id=b0104213af32435589ae19109b7aee7f]) successfully deleted
I20251026 02:56:41.838115 3643 raft_consensus.cc:697] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:41.838346 3643 consensus_queue.cc:237] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.843047 3587 raft_consensus.cc:515] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.844277 3587 leader_election.cc:290] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.846323 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c51b98a0c9e54daf9d1e457f34d56475" } modify_external_catalogs: true
I20251026 02:56:41.846549 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c51b98a0c9e54daf9d1e457f34d56475" } modify_external_catalogs: true
I20251026 02:56:41.848655 2372 tablet_replica.cc:333] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:41.849152 2372 raft_consensus.cc:2243] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.849858 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a4dc7d3d1de04432881624228345e98b" } new_table_name: "default.Table_85a72fb0164a4922a649a43010b4998a" modify_external_catalogs: true
I20251026 02:56:41.850234 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1eca935882b04878bb1362a022e82ecf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.850754 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1eca935882b04878bb1362a022e82ecf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.851058 2372 raft_consensus.cc:2272] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.851284 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e6801c439a1b498c82d443a4e2fc8e74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:41.851639 2372 ts_tablet_manager.cc:1916] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:41.851794 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6801c439a1b498c82d443a4e2fc8e74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.850708 2375 ts_tablet_manager.cc:1434] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.021s user 0.001s sys 0.002s
I20251026 02:56:41.852530 2375 tablet_bootstrap.cc:492] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:41.855944 2375 tablet_bootstrap.cc:654] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.857568 3647 raft_consensus.cc:515] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.858479 3647 leader_election.cc:290] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.860900 3372 raft_consensus.cc:493] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.861037 3372 raft_consensus.cc:515] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.861629 3372 leader_election.cc:290] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:41.862834 1916 leader_election.cc:343] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ae95654d129441e08fcb3f3b355ffc22
W20251026 02:56:41.864634 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 122c81787d5c43d28b1fb639e3542c76 consensus state has no leader
I20251026 02:56:41.865073 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0c717ffb0f0440c3a035bb9717943e08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.865789 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c717ffb0f0440c3a035bb9717943e08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.871222 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_00a6c982df6b4e6892bc060e529d5ee4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.871784 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00a6c982df6b4e6892bc060e529d5ee4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.874706 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_464e60af830f4416af8fdcf84f660a49" } modify_external_catalogs: true
I20251026 02:56:41.874975 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_464e60af830f4416af8fdcf84f660a49" } modify_external_catalogs: true
W20251026 02:56:41.876482 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 44874.3us
W20251026 02:56:41.876792 2053 leader_election.cc:343] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.876978 2053 leader_election.cc:304] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.877318 3647 raft_consensus.cc:2749] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:41.877774 2053 leader_election.cc:343] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.877943 2053 leader_election.cc:304] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.878226 3647 raft_consensus.cc:2749] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:41.878973 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8799a5af2b42cc92ccc0195cff98e7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:41.881306 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_067a40b223504137ad333312ad69073a" } new_table_name: "default.Table_853ff6e087ac44d8b3ea6ecb653d0a6e" modify_external_catalogs: true
I20251026 02:56:41.881443 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1ca5b8379c94deab1130bfe8bdc87b0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:41.888309 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_27c4161c9b184347971d3035b388e746" } new_table_name: "default.Table_7b365f1cd4bd422d9bdafd46408f8b27" modify_external_catalogs: true
W20251026 02:56:41.892320 2053 leader_election.cc:343] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:41.892668 2053 leader_election.cc:343] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:41.849400 2381 tablet_bootstrap.cc:654] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.862303 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae95654d129441e08fcb3f3b355ffc22" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.880048 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8799a5af2b42cc92ccc0195cff98e7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:41.894050 2259 raft_consensus.cc:3060] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.894224 1915 leader_election.cc:304] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:41.894673 3372 raft_consensus.cc:2749] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:41.882280 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1ca5b8379c94deab1130bfe8bdc87b0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:41.896432 2257 raft_consensus.cc:3060] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.899082 2257 raft_consensus.cc:2468] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
W20251026 02:56:41.899158 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be60e04675474c1ea4a69506bc35e68a consensus state has no leader
I20251026 02:56:41.899556 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0ea323ca655f45aa94b47eab713ebbd0" } modify_external_catalogs: true
I20251026 02:56:41.899588 2053 leader_election.cc:304] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.899741 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0ea323ca655f45aa94b47eab713ebbd0" } modify_external_catalogs: true
I20251026 02:56:41.900223 3647 raft_consensus.cc:2804] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 02:56:41.900314 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d85848daf97044e980060490a145400c consensus state has no leader
I20251026 02:56:41.900349 3647 raft_consensus.cc:697] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:41.900596 3647 consensus_queue.cc:237] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.900686 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_016a71d765444bc78bf9e3cf36571b0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.901316 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_016a71d765444bc78bf9e3cf36571b0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.902031 2259 raft_consensus.cc:2468] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:41.903215 2053 leader_election.cc:304] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:41.904119 3643 raft_consensus.cc:2804] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:41.904294 3643 raft_consensus.cc:697] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:41.904549 3643 consensus_queue.cc:237] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.909934 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2939f4b1afea4a4b8bbfe83ba4d58558"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:41.910786 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2939f4b1afea4a4b8bbfe83ba4d58558 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:41.913533 2195 consensus_peers.cc:597] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfc3aed64ad0439cb22fd6815c973d87. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:41.914986 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 607) took 8015 ms. Trace:
I20251026 02:56:41.915172 2377 rpcz_store.cc:276] 1026 02:56:33.899614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.899684 (+ 70us) service_pool.cc:225] Handling call
1026 02:56:41.803338 (+7903654us) ts_tablet_manager.cc:1151] Deleting tablet 358e805370ef410e9d65ee92a8d04f34
1026 02:56:41.803348 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.822249 (+ 18901us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.826131 (+ 3882us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.914967 (+ 88836us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7903440}
I20251026 02:56:41.915795 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 358e805370ef410e9d65ee92a8d04f34 (table default.Table_71583d73a49142ae9fec31193fb53542 [id=e86d9fc892a8409590c555e25b27a12f]) successfully deleted
I20251026 02:56:41.917843 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 608) took 8018 ms. Trace:
I20251026 02:56:41.918004 2377 rpcz_store.cc:276] 1026 02:56:33.899790 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.901086 (+ 1296us) service_pool.cc:225] Handling call
1026 02:56:41.915589 (+8014503us) ts_tablet_manager.cc:1151] Deleting tablet 9b0406ceb60b48969ca84f6268b43dbb
1026 02:56:41.917723 (+ 2134us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.917825 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8014319}
I20251026 02:56:41.918499 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 615) took 7989 ms. Trace:
W20251026 02:56:41.918681 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 9b0406ceb60b48969ca84f6268b43dbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b0406ceb60b48969ca84f6268b43dbb already in progress: creating tablet
I20251026 02:56:41.918849 2377 rpcz_store.cc:276] 1026 02:56:33.929105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.980798 (+ 51693us) service_pool.cc:225] Handling call
1026 02:56:41.918386 (+7937588us) ts_tablet_manager.cc:1151] Deleting tablet d212170f056a44dfb15cfcb7f9fbae2f
1026 02:56:41.918395 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.918485 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7937380}
I20251026 02:56:41.919782 2377 tablet_replica.cc:333] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539: stopping tablet replica
W20251026 02:56:41.920128 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet d212170f056a44dfb15cfcb7f9fbae2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d212170f056a44dfb15cfcb7f9fbae2f already in progress: creating tablet
I20251026 02:56:41.920204 2377 raft_consensus.cc:2243] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.920385 2377 raft_consensus.cc:2272] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.927959 2377 ts_tablet_manager.cc:1916] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.929388 1978 raft_consensus.cc:3060] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:41.933017 2381 tablet_bootstrap.cc:492] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:41.933254 2381 ts_tablet_manager.cc:1403] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.110s user 0.001s sys 0.005s
I20251026 02:56:41.934192 2381 raft_consensus.cc:359] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.934476 2381 raft_consensus.cc:385] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.934566 2381 raft_consensus.cc:740] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.934837 2381 consensus_queue.cc:260] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.935410 2381 ts_tablet_manager.cc:1434] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:41.935993 2381 tablet_bootstrap.cc:492] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:41.936935 2378 tablet_bootstrap.cc:654] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.937608 2381 tablet_bootstrap.cc:654] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.938853 2375 tablet_bootstrap.cc:492] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:41.938903 3372 raft_consensus.cc:493] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:41.939042 3372 raft_consensus.cc:515] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.939677 3372 leader_election.cc:290] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:41.940675 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a3491b303f94fe2a2c838297f73d464" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:41.940635 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a3491b303f94fe2a2c838297f73d464" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:41.942749 2375 ts_tablet_manager.cc:1403] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.001s
I20251026 02:56:41.943181 1916 leader_election.cc:304] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:41.943485 2375 raft_consensus.cc:359] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.943708 3372 raft_consensus.cc:2749] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:41.946153 2375 raft_consensus.cc:385] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.946303 2375 raft_consensus.cc:740] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.946594 2375 consensus_queue.cc:260] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.947316 2375 ts_tablet_manager.cc:1434] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251026 02:56:41.947634 2375 tablet_bootstrap.cc:492] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:41.948058 2372 ts_tablet_manager.cc:1929] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.949137 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3478623ef36e4ef7910a71260ba0ecb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:41.967097 2259 raft_consensus.cc:1275] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Refusing update from remote peer 0b6b0a6c4e144982b67980118cddba48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251026 02:56:41.967396 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3478623ef36e4ef7910a71260ba0ecb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:41.972584 2381 tablet_bootstrap.cc:492] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:41.972801 2381 ts_tablet_manager.cc:1403] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.001s
I20251026 02:56:41.973574 2381 raft_consensus.cc:359] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.973832 2381 raft_consensus.cc:385] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.973917 2381 raft_consensus.cc:740] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.974162 2381 consensus_queue.cc:260] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:41.976958 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7d238b86df1246ffa4b5d4bb9af0f387" } modify_external_catalogs: true
I20251026 02:56:41.977046 2381 ts_tablet_manager.cc:1434] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251026 02:56:41.977260 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7d238b86df1246ffa4b5d4bb9af0f387" } modify_external_catalogs: true
I20251026 02:56:41.977334 2381 tablet_bootstrap.cc:492] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:41.982417 2377 ts_tablet_manager.cc:1929] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:41.982626 2377 log.cc:1199] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7062c5ff91e14369ab69a2d199585c55
I20251026 02:56:41.983230 2377 ts_tablet_manager.cc:1950] T 7062c5ff91e14369ab69a2d199585c55 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:41.979027 2378 tablet_bootstrap.cc:492] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:41.984750 2378 ts_tablet_manager.cc:1403] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.159s user 0.002s sys 0.002s
I20251026 02:56:41.985733 2378 raft_consensus.cc:359] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.985885 2381 tablet_bootstrap.cc:654] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.985916 2378 raft_consensus.cc:385] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:41.985977 2378 raft_consensus.cc:740] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:41.986227 2378 consensus_queue.cc:260] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:41.989034 2378 ts_tablet_manager.cc:1434] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:41.989427 2378 tablet_bootstrap.cc:492] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:41.991400 2378 tablet_bootstrap.cc:654] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:41.963565 2376 ts_tablet_manager.cc:1929] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:41.992548 2376 log.cc:1199] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/a08efd37968846d2958e54831488235c
I20251026 02:56:41.992975 2376 ts_tablet_manager.cc:1950] T a08efd37968846d2958e54831488235c P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:41.993315 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 712) took 4057 ms. Trace:
I20251026 02:56:41.993450 2376 rpcz_store.cc:276] 1026 02:56:37.935614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.935684 (+ 70us) service_pool.cc:225] Handling call
1026 02:56:41.807222 (+3871538us) ts_tablet_manager.cc:1151] Deleting tablet a08efd37968846d2958e54831488235c
1026 02:56:41.807231 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.934180 (+126949us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.963455 (+ 29275us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.993299 (+ 29844us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3871334}
I20251026 02:56:41.994052 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet a08efd37968846d2958e54831488235c (table default.Table_80ca368a0ade4ac2b090b31b2ab56f12 [id=49dd7ccfa92946bebeaa4c178138f75a]) successfully deleted
I20251026 02:56:41.994540 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 711) took 4061 ms. Trace:
I20251026 02:56:41.994675 2376 rpcz_store.cc:276] 1026 02:56:37.932811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.937774 (+ 4963us) service_pool.cc:225] Handling call
1026 02:56:41.994415 (+4056641us) ts_tablet_manager.cc:1151] Deleting tablet 3aa7a90a29ec430286786bf8aa9d4c02
1026 02:56:41.994424 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.994523 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4056389}
W20251026 02:56:41.995229 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3aa7a90a29ec430286786bf8aa9d4c02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3aa7a90a29ec430286786bf8aa9d4c02 already in progress: creating tablet
I20251026 02:56:41.995601 2376 tablet_replica.cc:333] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:41.995836 2376 raft_consensus.cc:2243] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:41.996042 2376 raft_consensus.cc:2272] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:41.996644 2376 ts_tablet_manager.cc:1916] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:41.949126 2375 tablet_bootstrap.cc:654] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:42.008536 3647 raft_consensus.cc:493] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:41.967020 2053 consensus_peers.cc:597] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:42.008692 3647 raft_consensus.cc:515] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.009305 3647 leader_election.cc:290] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:42.009454 3647 consensus_queue.cc:1048] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:42.010210 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cf76ef8f6af445eb0ce15c8d79136c5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:42.011758 2053 leader_election.cc:343] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.012091 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cf76ef8f6af445eb0ce15c8d79136c5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:42.012796 2053 leader_election.cc:343] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.012902 2053 leader_election.cc:304] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.013260 3647 raft_consensus.cc:2749] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:42.018096 3372 raft_consensus.cc:493] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.018256 3372 raft_consensus.cc:515] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:42.018875 3372 leader_election.cc:290] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.019480 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85f9fcae83a04c6e907c7e9c8cd95dbe" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:42.020668 3622 raft_consensus.cc:493] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.020767 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85f9fcae83a04c6e907c7e9c8cd95dbe" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:42.020799 3622 raft_consensus.cc:515] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.021440 3622 leader_election.cc:290] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.022341 1915 leader_election.cc:304] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:42.022539 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfa68f2c942c43e2a02e9d333bdfa2a2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:42.022701 2118 raft_consensus.cc:2468] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:42.022984 3372 raft_consensus.cc:2749] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:42.023289 2195 leader_election.cc:304] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:42.023608 3622 raft_consensus.cc:2804] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:42.023689 3622 raft_consensus.cc:493] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:42.023763 3622 raft_consensus.cc:3060] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:42.026453 3622 raft_consensus.cc:515] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.027457 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfa68f2c942c43e2a02e9d333bdfa2a2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:42.027632 2118 raft_consensus.cc:3060] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:42.028229 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfa68f2c942c43e2a02e9d333bdfa2a2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:42.028820 2194 leader_election.cc:343] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.030239 2118 raft_consensus.cc:2468] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:42.030558 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfa68f2c942c43e2a02e9d333bdfa2a2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:42.033581 3754 tablet.cc:1722] T 8862cd48a5744c99b32f365cb77fa7c4 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:42.039774 2195 leader_election.cc:304] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:42.040033 3517 raft_consensus.cc:2804] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:42.040162 3517 raft_consensus.cc:697] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:42.040410 3517 consensus_queue.cc:237] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.041257 3689 mvcc.cc:204] Tried to move back new op lower bound from 7214888558449201152 to 7214888554916450304. Current Snapshot: MvccSnapshot[applied={T|T < 7214888558449201152}]
I20251026 02:56:42.041489 3750 tablet.cc:1722] T 8862cd48a5744c99b32f365cb77fa7c4 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:42.044934 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 614) took 8144 ms. Trace:
I20251026 02:56:42.045184 3622 leader_election.cc:290] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.045256 2377 rpcz_store.cc:276] 1026 02:56:33.900383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:33.980419 (+ 80036us) service_pool.cc:225] Handling call
1026 02:56:41.919737 (+7939318us) ts_tablet_manager.cc:1151] Deleting tablet 7062c5ff91e14369ab69a2d199585c55
1026 02:56:41.919748 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.961515 (+ 41767us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.982309 (+ 20794us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.044911 (+ 62602us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7812445}
I20251026 02:56:42.045708 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7062c5ff91e14369ab69a2d199585c55 (table default.Table_d48e174e391f42d6aa0b05c322b5b828 [id=91a568329da84558b31575e5127f873f]) successfully deleted
I20251026 02:56:42.054466 2372 log.cc:1199] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/741792cf96164695b03de3fb86b12602
I20251026 02:56:42.055127 2372 ts_tablet_manager.cc:1950] T 741792cf96164695b03de3fb86b12602 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:42.056255 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 616) took 7731 ms. Trace:
I20251026 02:56:42.056355 2377 rpcz_store.cc:276] 1026 02:56:34.327182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.331540 (+ 4358us) service_pool.cc:225] Handling call
1026 02:56:42.046032 (+7714492us) ts_tablet_manager.cc:1151] Deleting tablet 82e4bc6195484074b0da610a8d1b65bc
1026 02:56:42.056087 (+ 10055us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.056231 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7714314}
I20251026 02:56:42.056777 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 630) took 7431 ms. Trace:
I20251026 02:56:42.056856 2377 rpcz_store.cc:276] 1026 02:56:34.626401 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.626720 (+ 319us) service_pool.cc:225] Handling call
1026 02:56:42.056653 (+7429933us) ts_tablet_manager.cc:1151] Deleting tablet 6699b50d87964f6a89c365171253ad25
1026 02:56:42.056667 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.056762 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7256804}
I20251026 02:56:42.057229 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 631) took 7429 ms. Trace:
I20251026 02:56:42.057299 2377 rpcz_store.cc:276] 1026 02:56:34.629106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.629717 (+ 611us) service_pool.cc:225] Handling call
1026 02:56:42.057133 (+7427416us) ts_tablet_manager.cc:1151] Deleting tablet 5824831b8e184daf8b40543686c06467
1026 02:56:42.057141 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.057215 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7252815}
I20251026 02:56:42.057704 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 632) took 7426 ms. Trace:
I20251026 02:56:42.057839 2377 rpcz_store.cc:276] 1026 02:56:34.650109 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.652386 (+ 2277us) service_pool.cc:225] Handling call
1026 02:56:42.057590 (+7405204us) ts_tablet_manager.cc:1151] Deleting tablet 848e414f6e134761818039f7e98500ac
1026 02:56:42.057597 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.057686 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7252501}
I20251026 02:56:42.058444 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 635) took 7364 ms. Trace:
I20251026 02:56:42.058612 2377 rpcz_store.cc:276] 1026 02:56:34.694396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.694951 (+ 555us) service_pool.cc:225] Handling call
1026 02:56:42.058246 (+7363295us) ts_tablet_manager.cc:1151] Deleting tablet 5ddc7972f29a429ea51f6c6c1a074baf
1026 02:56:42.058258 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.058402 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7237902}
I20251026 02:56:42.059026 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 657) took 7101 ms. Trace:
I20251026 02:56:42.059242 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 646) took 7363 ms. Trace:
I20251026 02:56:42.059391 2377 rpcz_store.cc:276] 1026 02:56:34.695357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.889759 (+194402us) service_pool.cc:225] Handling call
1026 02:56:42.059005 (+7169246us) ts_tablet_manager.cc:1151] Deleting tablet c09153fbfe18434393c188058a3461d5
1026 02:56:42.059042 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.059193 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7145555}
I20251026 02:56:42.059791 2377 tablet_replica.cc:333] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:42.060158 2377 raft_consensus.cc:2243] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:42.062527 2381 tablet_bootstrap.cc:492] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:42.062737 2381 ts_tablet_manager.cc:1403] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.085s user 0.018s sys 0.022s
I20251026 02:56:42.063645 2372 rpcz_store.cc:276] 1026 02:56:34.957389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.957435 (+ 46us) service_pool.cc:225] Handling call
1026 02:56:41.848588 (+6891153us) ts_tablet_manager.cc:1151] Deleting tablet 741792cf96164695b03de3fb86b12602
1026 02:56:41.848604 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:41.944820 (+ 96216us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:41.947930 (+ 3110us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.059004 (+111074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6858527}
I20251026 02:56:42.063648 2377 raft_consensus.cc:2272] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.067514 3372 raft_consensus.cc:493] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.067678 3372 raft_consensus.cc:515] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:42.067780 2378 tablet_bootstrap.cc:492] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:42.067919 2378 ts_tablet_manager.cc:1403] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.002s
I20251026 02:56:42.068373 3372 leader_election.cc:290] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.068488 2372 tablet_replica.cc:333] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:42.068665 2372 raft_consensus.cc:2243] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.068817 2372 raft_consensus.cc:2272] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:42.069334 1916 leader_election.cc:343] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 276bd3b97c97426daf802e1f57feb265
W20251026 02:56:42.070183 1915 leader_election.cc:343] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 276bd3b97c97426daf802e1f57feb265
I20251026 02:56:42.068131 2381 raft_consensus.cc:359] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.070286 1915 leader_election.cc:304] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:42.070618 2377 ts_tablet_manager.cc:1916] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.070748 3372 raft_consensus.cc:2749] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:42.070303 2381 raft_consensus.cc:385] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.071652 2378 raft_consensus.cc:359] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:42.071803 2378 raft_consensus.cc:385] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.071849 2378 raft_consensus.cc:740] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.071171 2381 raft_consensus.cc:740] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.072037 2378 consensus_queue.cc:260] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:42.073482 2027 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:42.073597 2381 consensus_queue.cc:260] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.069381 2372 ts_tablet_manager.cc:1916] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.073419 2375 tablet_bootstrap.cc:492] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:42.079766 2381 ts_tablet_manager.cc:1434] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251026 02:56:42.080408 2375 ts_tablet_manager.cc:1403] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.133s user 0.005s sys 0.005s
I20251026 02:56:42.081202 2375 raft_consensus.cc:359] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.081360 2375 raft_consensus.cc:385] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.081420 2375 raft_consensus.cc:740] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.081645 2375 consensus_queue.cc:260] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.082204 2375 ts_tablet_manager.cc:1434] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:42.082728 2375 tablet_bootstrap.cc:492] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:42.084437 2375 tablet_bootstrap.cc:654] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:42.091888 2194 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 63069.3us
I20251026 02:56:42.092885 2381 tablet_bootstrap.cc:492] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:42.094267 2381 tablet_bootstrap.cc:654] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:42.096886 2376 ts_tablet_manager.cc:1929] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.103447 3777 raft_consensus.cc:493] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.103617 3777 raft_consensus.cc:515] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.104100 2381 tablet_bootstrap.cc:492] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:42.104262 2381 ts_tablet_manager.cc:1403] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20251026 02:56:42.104288 3777 leader_election.cc:290] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:42.105027 2381 raft_consensus.cc:359] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.105202 2381 raft_consensus.cc:385] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.105268 2381 raft_consensus.cc:740] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.105445 2381 consensus_queue.cc:260] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.105693 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5228f482c0b4fcbbdf8ebc31bb62819" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:42.105901 2381 ts_tablet_manager.cc:1434] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:42.106262 2381 tablet_bootstrap.cc:492] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:42.106405 2053 leader_election.cc:343] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.106647 3777 raft_consensus.cc:493] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.106758 3777 raft_consensus.cc:515] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.107298 3777 leader_election.cc:290] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:42.107501 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 82e4bc6195484074b0da610a8d1b65bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82e4bc6195484074b0da610a8d1b65bc already in progress: creating tablet
I20251026 02:56:42.107622 2381 tablet_bootstrap.cc:654] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:42.107693 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 50817.3us
W20251026 02:56:42.110692 2194 leader_election.cc:343] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.105216 2377 ts_tablet_manager.cc:1929] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.111172 2377 log.cc:1199] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/efd751ee4b1d433198dcd00c081f648d
I20251026 02:56:42.111647 2377 ts_tablet_manager.cc:1950] T efd751ee4b1d433198dcd00c081f648d P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:42.114064 2372 ts_tablet_manager.cc:1929] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.116760 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23062e7292f44ad5ad9dfc22a401741c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:42.118108 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 648) took 7422 ms. Trace:
I20251026 02:56:42.118260 2377 rpcz_store.cc:276] 1026 02:56:34.695534 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.900133 (+204599us) service_pool.cc:225] Handling call
1026 02:56:42.059719 (+7159586us) ts_tablet_manager.cc:1151] Deleting tablet efd751ee4b1d433198dcd00c081f648d
1026 02:56:42.059728 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.090755 (+ 31027us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.104934 (+ 14179us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.118088 (+ 13154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4785,"tablet-delete.queue_time_us":7145741}
I20251026 02:56:42.119715 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 674) took 7195 ms. Trace:
I20251026 02:56:42.120111 2377 rpcz_store.cc:276] 1026 02:56:34.924578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.178689 (+254111us) service_pool.cc:225] Handling call
1026 02:56:42.119604 (+6940915us) ts_tablet_manager.cc:1151] Deleting tablet a74d6a4e041347578b0976a8b54bf22c
1026 02:56:42.119613 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.119699 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6940739}
I20251026 02:56:42.120599 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 675) took 7195 ms. Trace:
I20251026 02:56:42.120965 2377 rpcz_store.cc:276] 1026 02:56:34.924671 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.178876 (+254205us) service_pool.cc:225] Handling call
1026 02:56:42.120505 (+6941629us) ts_tablet_manager.cc:1151] Deleting tablet 657e07d0b17c44d4b78b97210f17ce40
1026 02:56:42.120514 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.120587 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6941533}
I20251026 02:56:42.121300 2376 log.cc:1199] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/5caeb11322fe48348ebb523967152187
I20251026 02:56:42.121800 2376 ts_tablet_manager.cc:1950] T 5caeb11322fe48348ebb523967152187 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:42.122547 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 713) took 4159 ms. Trace:
I20251026 02:56:42.122670 2376 rpcz_store.cc:276] 1026 02:56:37.963232 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.963281 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:41.995555 (+4032274us) ts_tablet_manager.cc:1151] Deleting tablet 5caeb11322fe48348ebb523967152187
1026 02:56:41.995564 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.093767 (+ 98203us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.096783 (+ 3016us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.122532 (+ 25749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4032087}
I20251026 02:56:42.123322 2376 tablet_replica.cc:333] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:42.123569 2376 raft_consensus.cc:2243] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.105263 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5228f482c0b4fcbbdf8ebc31bb62819" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:42.123759 2376 raft_consensus.cc:2272] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.124295 2376 ts_tablet_manager.cc:1916] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:42.119321 2053 leader_election.cc:343] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:42.124711 2053 leader_election.cc:343] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.124814 2053 leader_election.cc:304] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.125208 3778 raft_consensus.cc:2749] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:42.119503 3777 raft_consensus.cc:493] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.125497 3777 raft_consensus.cc:515] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.126041 3777 leader_election.cc:290] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:42.130228 3239 raft_consensus.cc:493] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.130374 3239 raft_consensus.cc:515] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.130698 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99d318aa0d1847c58167bf0af9e75aa2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:42.130928 3239 leader_election.cc:290] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.131302 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 676) took 7206 ms. Trace:
W20251026 02:56:42.131664 2053 leader_election.cc:343] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.131609 2377 rpcz_store.cc:276] 1026 02:56:34.924747 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.178982 (+254235us) service_pool.cc:225] Handling call
1026 02:56:42.121208 (+6942226us) ts_tablet_manager.cc:1151] Deleting tablet 3efafaac6c634d088396d56c8debd990
1026 02:56:42.128834 (+ 7626us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.131198 (+ 2364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6942075}
I20251026 02:56:42.136427 2372 log.cc:1199] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/d65b0d79e2e54af9810022b683c3288b
I20251026 02:56:42.136955 2372 ts_tablet_manager.cc:1950] T d65b0d79e2e54af9810022b683c3288b P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:42.116293 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23062e7292f44ad5ad9dfc22a401741c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:42.119434 1980 raft_consensus.cc:3060] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:42.126749 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99d318aa0d1847c58167bf0af9e75aa2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:42.143898 3517 consensus_queue.cc:1048] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:42.132462 2194 consensus_peers.cc:597] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:42.107090 2378 ts_tablet_manager.cc:1434] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20251026 02:56:42.146061 2378 tablet_bootstrap.cc:492] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:42.132390 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882cd1680aa140f6a63a961cd4aec25a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:42.147226 2194 leader_election.cc:343] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.148489 2378 tablet_bootstrap.cc:654] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:42.144795 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 677) took 7158 ms. Trace:
I20251026 02:56:42.149215 2377 rpcz_store.cc:276] 1026 02:56:34.985874 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.179147 (+193273us) service_pool.cc:225] Handling call
1026 02:56:42.143755 (+6964608us) ts_tablet_manager.cc:1151] Deleting tablet 1219e8bdfde9444f9bfdf1d626389e88
1026 02:56:42.143980 (+ 225us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.144608 (+ 628us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5376,"tablet-delete.queue_time_us":6964456}
I20251026 02:56:42.149662 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 673) took 7225 ms. Trace:
I20251026 02:56:42.149742 2377 rpcz_store.cc:276] 1026 02:56:34.924485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.173905 (+249420us) service_pool.cc:225] Handling call
1026 02:56:42.149540 (+6975635us) ts_tablet_manager.cc:1151] Deleting tablet bc4ca4423e3a4e0c96846babec931fec
1026 02:56:42.149551 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.149646 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6838414}
I20251026 02:56:42.149979 2377 tablet_replica.cc:333] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:42.150177 2377 raft_consensus.cc:2243] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:42.150516 2377 raft_consensus.cc:2272] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.141453 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882cd1680aa140f6a63a961cd4aec25a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:42.144383 2053 leader_election.cc:343] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.153807 2053 leader_election.cc:304] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.153880 2117 raft_consensus.cc:2468] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:42.154312 2053 leader_election.cc:343] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.154435 2053 leader_election.cc:304] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.154685 3777 raft_consensus.cc:2749] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:42.154780 3778 raft_consensus.cc:2749] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:42.155072 2195 leader_election.cc:304] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.155653 3517 raft_consensus.cc:2804] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:42.155750 3517 raft_consensus.cc:493] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:42.155795 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 658) took 7198 ms. Trace:
I20251026 02:56:42.155825 3517 raft_consensus.cc:3060] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:42.155874 2372 rpcz_store.cc:276] 1026 02:56:34.957633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.957675 (+ 42us) service_pool.cc:225] Handling call
1026 02:56:42.068429 (+7110754us) ts_tablet_manager.cc:1151] Deleting tablet d65b0d79e2e54af9810022b683c3288b
1026 02:56:42.068443 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.079959 (+ 11516us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.113947 (+ 33988us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.155776 (+ 41829us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7078274}
I20251026 02:56:42.158667 2377 ts_tablet_manager.cc:1916] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.159884 2372 tablet_replica.cc:333] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:42.159916 3517 raft_consensus.cc:515] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.160424 2372 raft_consensus.cc:2243] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.161432 2372 raft_consensus.cc:2272] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.161826 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882cd1680aa140f6a63a961cd4aec25a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:42.161528 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882cd1680aa140f6a63a961cd4aec25a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:42.162285 2117 raft_consensus.cc:3060] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:42.162506 2194 leader_election.cc:343] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.163467 2372 ts_tablet_manager.cc:1916] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.167658 3517 leader_election.cc:290] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.171142 2378 tablet_bootstrap.cc:492] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:42.171327 2378 ts_tablet_manager.cc:1403] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20251026 02:56:42.172061 2378 raft_consensus.cc:359] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.172284 2378 raft_consensus.cc:385] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.172355 2378 raft_consensus.cc:740] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.172560 2378 consensus_queue.cc:260] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.173041 2378 ts_tablet_manager.cc:1434] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 02:56:42.173350 2378 tablet_bootstrap.cc:492] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:42.174994 2378 tablet_bootstrap.cc:654] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:42.176304 3517 raft_consensus.cc:493] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.176487 3517 raft_consensus.cc:515] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:42.179876 2195 leader_election.cc:343] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 3327d26a2c2c4d0b86d98151f5940bd7
I20251026 02:56:42.183218 3517 leader_election.cc:290] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.187227 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3327d26a2c2c4d0b86d98151f5940bd7" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:42.187458 1982 raft_consensus.cc:2468] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:42.188933 3616 raft_consensus.cc:493] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.189069 3616 raft_consensus.cc:515] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.189628 3616 leader_election.cc:290] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.190115 3372 raft_consensus.cc:493] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.190244 3372 raft_consensus.cc:515] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:42.190809 3372 leader_election.cc:290] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.191637 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9cd10092aac40068a1c7f85e514fd37" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:42.192317 2194 leader_election.cc:304] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:42.192595 3616 raft_consensus.cc:2804] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 02:56:42.192670 3616 raft_consensus.cc:493] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:42.192764 3616 raft_consensus.cc:3060] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:42.192742 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 741792cf96164695b03de3fb86b12602 (table default.Table_d4304259ac684b8097cdbf5511ab67d7 [id=d64901838ed94621ad11692d9e0aa80e]) successfully deleted
W20251026 02:56:42.193395 1648 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 124312us
I20251026 02:56:42.194346 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet d65b0d79e2e54af9810022b683c3288b (table default.Table_bcac335a2c9a479fb5dea86b2aeffde6 [id=30a5783033cb4f7f929e766fb47d92ff]) successfully deleted
I20251026 02:56:42.195472 3616 raft_consensus.cc:515] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
W20251026 02:56:42.195704 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 6699b50d87964f6a89c365171253ad25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6699b50d87964f6a89c365171253ad25 already in progress: creating tablet
W20251026 02:56:42.195956 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 5824831b8e184daf8b40543686c06467 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5824831b8e184daf8b40543686c06467 already in progress: creating tablet
I20251026 02:56:42.196035 3616 leader_election.cc:290] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:42.196581 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 848e414f6e134761818039f7e98500ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 848e414f6e134761818039f7e98500ac already in progress: creating tablet
W20251026 02:56:42.196862 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 5ddc7972f29a429ea51f6c6c1a074baf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ddc7972f29a429ea51f6c6c1a074baf already in progress: creating tablet
W20251026 02:56:42.197086 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet c09153fbfe18434393c188058a3461d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c09153fbfe18434393c188058a3461d5 already in progress: creating tablet
W20251026 02:56:42.197315 2195 leader_election.cc:343] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 3327d26a2c2c4d0b86d98151f5940bd7
I20251026 02:56:42.197569 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet efd751ee4b1d433198dcd00c081f648d (table default.Table_c2ef795e86444f9f85bb7d0bcca51240 [id=20d9432d9f6443099d7fc0acb952a839]) successfully deleted
W20251026 02:56:42.197813 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet a74d6a4e041347578b0976a8b54bf22c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a74d6a4e041347578b0976a8b54bf22c already in progress: creating tablet
W20251026 02:56:42.198030 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 657e07d0b17c44d4b78b97210f17ce40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 657e07d0b17c44d4b78b97210f17ce40 already in progress: creating tablet
W20251026 02:56:42.198251 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3efafaac6c634d088396d56c8debd990 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3efafaac6c634d088396d56c8debd990 already in progress: creating tablet
W20251026 02:56:42.198448 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 1219e8bdfde9444f9bfdf1d626389e88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1219e8bdfde9444f9bfdf1d626389e88 already in progress: creating tablet
W20251026 02:56:42.198652 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet bc4ca4423e3a4e0c96846babec931fec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc4ca4423e3a4e0c96846babec931fec already in progress: creating tablet
I20251026 02:56:42.198905 3616 consensus_queue.cc:1048] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:42.199186 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 5caeb11322fe48348ebb523967152187 (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
I20251026 02:56:42.200757 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3327d26a2c2c4d0b86d98151f5940bd7" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:42.200943 1982 raft_consensus.cc:3060] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:42.203855 1982 raft_consensus.cc:2468] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 2.
I20251026 02:56:42.200918 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c31d2704d1f14ddead69c794cc631ec1" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:42.204913 2194 leader_election.cc:304] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:42.205088 2377 ts_tablet_manager.cc:1929] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.205431 3616 raft_consensus.cc:2804] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 02:56:42.205794 2377 log.cc:1199] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/30c619ad37364747b6ebf97a73ef45a2
I20251026 02:56:42.208166 2377 ts_tablet_manager.cc:1950] T 30c619ad37364747b6ebf97a73ef45a2 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:42.206015 3616 raft_consensus.cc:697] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 2 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:42.208762 3616 consensus_queue.cc:237] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:42.210449 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 689) took 6179 ms. Trace:
I20251026 02:56:42.210896 2377 rpcz_store.cc:276] 1026 02:56:36.031118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.031175 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:42.149938 (+6118763us) ts_tablet_manager.cc:1151] Deleting tablet 30c619ad37364747b6ebf97a73ef45a2
1026 02:56:42.149946 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.183543 (+ 33597us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.204154 (+ 20611us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.210273 (+ 6119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6118511}
I20251026 02:56:42.211860 3239 raft_consensus.cc:493] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.212245 3239 raft_consensus.cc:515] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.212762 3239 leader_election.cc:290] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.212925 1980 raft_consensus.cc:1275] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20251026 02:56:42.213380 2194 leader_election.cc:343] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.213589 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 682) took 6280 ms. Trace:
I20251026 02:56:42.214668 3239 consensus_queue.cc:1048] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:42.214093 2377 rpcz_store.cc:276] 1026 02:56:35.932950 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.933012 (+ 62us) service_pool.cc:225] Handling call
1026 02:56:42.212592 (+6279580us) ts_tablet_manager.cc:1151] Deleting tablet 902cc4f9dbbd4d8baa829448f2a4687a
1026 02:56:42.212602 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.213345 (+ 743us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6145671}
I20251026 02:56:42.215233 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 686) took 6280 ms. Trace:
I20251026 02:56:42.215308 2377 rpcz_store.cc:276] 1026 02:56:35.934738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.934787 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:42.215108 (+6280321us) ts_tablet_manager.cc:1151] Deleting tablet 1ffead7ca2d946c78aa4b8a602865fc8
1026 02:56:42.215118 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.215217 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6148028}
I20251026 02:56:42.215639 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 685) took 6281 ms. Trace:
I20251026 02:56:42.215709 2377 rpcz_store.cc:276] 1026 02:56:35.934499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.934544 (+ 45us) service_pool.cc:225] Handling call
1026 02:56:42.215548 (+6281004us) ts_tablet_manager.cc:1151] Deleting tablet 1dbbfb77851d443f8cf4d798f3e07e59
1026 02:56:42.215556 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.215626 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":736,"tablet-delete.queue_time_us":6148561}
I20251026 02:56:42.216123 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 688) took 6281 ms. Trace:
I20251026 02:56:42.216189 2377 rpcz_store.cc:276] 1026 02:56:35.935042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.935100 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:42.216037 (+6280937us) ts_tablet_manager.cc:1151] Deleting tablet 351cd60b076a49949f03640c981845c4
1026 02:56:42.216045 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.216109 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":743,"tablet-delete.queue_time_us":6148989}
I20251026 02:56:42.216531 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 687) took 6281 ms. Trace:
I20251026 02:56:42.216601 2377 rpcz_store.cc:276] 1026 02:56:35.934832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.934875 (+ 43us) service_pool.cc:225] Handling call
1026 02:56:42.216441 (+6281566us) ts_tablet_manager.cc:1151] Deleting tablet 24f16e40a30c41788a6f9813c235c5ab
1026 02:56:42.216449 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.216517 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":827,"tablet-delete.queue_time_us":6149389}
I20251026 02:56:42.216852 2377 tablet_replica.cc:333] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:42.217061 2377 raft_consensus.cc:2243] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:42.217391 2377 raft_consensus.cc:2272] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.217721 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 30c619ad37364747b6ebf97a73ef45a2 (table default.Table_ad4b4db01c7d4fe0a4e99f2f54cee0e8 [id=65a13a0482834ea6b52b71c953d61e5d]) successfully deleted
W20251026 02:56:42.218125 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 902cc4f9dbbd4d8baa829448f2a4687a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 902cc4f9dbbd4d8baa829448f2a4687a already in progress: creating tablet
W20251026 02:56:42.218429 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 1ffead7ca2d946c78aa4b8a602865fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ffead7ca2d946c78aa4b8a602865fc8 already in progress: creating tablet
W20251026 02:56:42.218688 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 1dbbfb77851d443f8cf4d798f3e07e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dbbfb77851d443f8cf4d798f3e07e59 already in progress: creating tablet
W20251026 02:56:42.218950 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 351cd60b076a49949f03640c981845c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 351cd60b076a49949f03640c981845c4 already in progress: creating tablet
W20251026 02:56:42.219208 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 24f16e40a30c41788a6f9813c235c5ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24f16e40a30c41788a6f9813c235c5ab already in progress: creating tablet
I20251026 02:56:42.219231 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7177931353164b2aa82c00b2807ccd63" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:42.227708 2194 leader_election.cc:343] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.240602 3537 raft_consensus.cc:493] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.240768 3537 raft_consensus.cc:515] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.241533 3537 leader_election.cc:290] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.251677 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c6e3e47b3e7421e945cfbe671b06abe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:42.251946 2114 raft_consensus.cc:2468] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:42.252415 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c6e3e47b3e7421e945cfbe671b06abe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:42.166884 2117 raft_consensus.cc:2468] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:42.191395 2107 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9cd10092aac40068a1c7f85e514fd37" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:42.254112 1916 leader_election.cc:304] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:42.191615 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c31d2704d1f14ddead69c794cc631ec1" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:42.217222 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7177931353164b2aa82c00b2807ccd63" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:42.254660 2116 raft_consensus.cc:2468] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:42.254729 3372 raft_consensus.cc:2749] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:42.254660 2118 raft_consensus.cc:2468] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:42.258157 2195 leader_election.cc:304] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:42.258684 2195 leader_election.cc:304] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.258991 3239 raft_consensus.cc:2804] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:42.259090 3239 raft_consensus.cc:493] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:42.259148 2195 leader_election.cc:304] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.259158 3239 raft_consensus.cc:3060] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:42.259652 3537 raft_consensus.cc:2804] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:42.259732 3537 raft_consensus.cc:493] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:42.259804 3537 raft_consensus.cc:3060] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:42.262290 3239 raft_consensus.cc:515] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.262975 3239 leader_election.cc:290] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.263190 3239 raft_consensus.cc:2804] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:42.263303 3239 raft_consensus.cc:493] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:42.263410 3239 raft_consensus.cc:3060] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:42.263443 3537 raft_consensus.cc:515] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.264415 3537 leader_election.cc:290] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.266058 3239 raft_consensus.cc:515] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.266644 3239 leader_election.cc:290] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.267777 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c6e3e47b3e7421e945cfbe671b06abe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:42.268107 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7177931353164b2aa82c00b2807ccd63" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:42.268683 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c31d2704d1f14ddead69c794cc631ec1" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:42.269047 2194 leader_election.cc:343] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.274749 1658 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe (table default.Table_47994972d3e0469c9acc96120aad4e1f [id=87349d9c6c764382b828e716e7854c59]). Expected version 3 got 0
I20251026 02:56:42.275218 2375 tablet_bootstrap.cc:492] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:42.275230 1658 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 23062e7292f44ad5ad9dfc22a401741c (table default.Table_11cf708e0e8243c887777e42acc25490 [id=2e17bae10e0b48a492f5ee0bed7d00d0]). Expected version 1 got 0
I20251026 02:56:42.275547 1661 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_e6b19dda39ad45baae642c27b07db42d [id=cba6d4517d244d7580e525729e871745] alter complete (version 4)
I20251026 02:56:42.276299 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 1219e8bdfde9444f9bfdf1d626389e88 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:42.276943 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3c1d783b88b0493f8aac8545f4cb3049"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:42.277170 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_35a36ec0f72d4af79182e095d6788ca4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:42.277469 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_df21327a17374d5f857a5f55cc511c52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:42.277593 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c1d783b88b0493f8aac8545f4cb3049 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:42.277684 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35a36ec0f72d4af79182e095d6788ca4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:42.277899 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df21327a17374d5f857a5f55cc511c52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:42.278993 2375 ts_tablet_manager.cc:1403] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.196s user 0.000s sys 0.004s
I20251026 02:56:42.281054 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3050e72d42417ab58a05f7e3aa56b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.281601 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet ae95654d129441e08fcb3f3b355ffc22 with delete_type TABLET_DATA_DELETED (replaced by tablet c2d6691bb4cf4a8292b0c082e54a91a5) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.282073 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b7d85a2941545ceaa16b22e8328b4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.282534 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet d65b0d79e2e54af9810022b683c3288b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.284593 2377 ts_tablet_manager.cc:1916] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.300470 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b083b1c076f1492a93b6e8f74e8bc574" } new_table_name: "default.Table_6681b8f0b9c24bdabf0540b3e6c7d98f" modify_external_catalogs: true
W20251026 02:56:42.301955 2194 leader_election.cc:343] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:42.302207 2194 leader_election.cc:343] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:42.302400 2194 leader_election.cc:343] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.285540 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c7efb70c0804c149f36440a4a9e6ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:42.285581 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet ae95654d129441e08fcb3f3b355ffc22 with delete_type TABLET_DATA_DELETED (replaced by tablet c2d6691bb4cf4a8292b0c082e54a91a5) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:42.285616 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 34c6a7c774044411a4733b115080c4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:42.280187 2375 raft_consensus.cc:359] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.304503 2375 raft_consensus.cc:385] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.304600 2375 raft_consensus.cc:740] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.304893 2375 consensus_queue.cc:260] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.305620 2375 ts_tablet_manager.cc:1434] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20251026 02:56:42.305986 2375 tablet_bootstrap.cc:492] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:42.307741 2375 tablet_bootstrap.cc:654] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:42.313685 2372 ts_tablet_manager.cc:1929] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.315246 2376 ts_tablet_manager.cc:1929] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:42.315402 2376 log.cc:1199] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/276bd3b97c97426daf802e1f57feb265
I20251026 02:56:42.315707 2376 ts_tablet_manager.cc:1950] T 276bd3b97c97426daf802e1f57feb265 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
W20251026 02:56:42.321254 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 66ce026fce6e432faaf5f334c8af3f63 consensus state has no leader
I20251026 02:56:42.321609 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet a4f4d7f87916425999574fd5b9f2fdf7 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:42.322471 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f41eab2536d54c839327b1084896bba4" } new_table_name: "default.Table_49ed105063f845bd92ee37a1ce8d97a7" modify_external_catalogs: true
I20251026 02:56:42.327877 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 714) took 4295 ms. Trace:
I20251026 02:56:42.328024 2376 rpcz_store.cc:276] 1026 02:56:38.032738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.032796 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:42.123279 (+4090483us) ts_tablet_manager.cc:1151] Deleting tablet 276bd3b97c97426daf802e1f57feb265
1026 02:56:42.123288 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.257918 (+134630us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.315145 (+ 57227us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.327288 (+ 12143us) spinlock_profiling.cc:243] Waited 6.16 ms on lock 0x61800000d518. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc41a1782fd 00007fc41a17936e 00007fc42ddabc9d 00007fc42dd2a203 00007fc429f5f6f3 00007fc429f922d4 00007fc429f72d70 00007fc429f72afd 0000000000a8867c 00007fc41a1d5d7f
1026 02:56:42.327849 (+ 561us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14170496,"tablet-delete.queue_time_us":4090276}
I20251026 02:56:42.329084 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 276bd3b97c97426daf802e1f57feb265 (table default.Table_3babf04294cc48ef92df6ffe8ab43e5a [id=e91358e10bab4589a6f4e815a85a5f7d]) successfully deleted
I20251026 02:56:42.331413 2377 ts_tablet_manager.cc:1929] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:42.333889 2376 tablet_replica.cc:333] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:42.334247 2376 raft_consensus.cc:2243] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.334472 2376 raft_consensus.cc:2272] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.328351 3778 raft_consensus.cc:493] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.334661 3778 raft_consensus.cc:515] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.335299 3778 leader_election.cc:290] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:42.333671 2375 tablet_bootstrap.cc:492] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:42.335737 2375 ts_tablet_manager.cc:1403] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.003s
W20251026 02:56:42.336091 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 662f2f0dfc804ddbbc4b71118a71d048 consensus state has no leader
I20251026 02:56:42.337524 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a53aedec98894542aa3c419b2295c67d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:42.337939 2372 log.cc:1199] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3b7d85a2941545ceaa16b22e8328b4ff
I20251026 02:56:42.338362 2372 ts_tablet_manager.cc:1950] T 3b7d85a2941545ceaa16b22e8328b4ff P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:42.336876 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c6a7c774044411a4733b115080c4c6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:42.339021 2376 ts_tablet_manager.cc:1916] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:42.339061 2053 leader_election.cc:343] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.340137 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 659) took 7382 ms. Trace:
I20251026 02:56:42.340797 2377 log.cc:1199] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/865fbd674db34f668fb5a2f5836a9b43
I20251026 02:56:42.343863 2372 rpcz_store.cc:276] 1026 02:56:34.957848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:34.957908 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:42.159822 (+7201914us) ts_tablet_manager.cc:1151] Deleting tablet 3b7d85a2941545ceaa16b22e8328b4ff
1026 02:56:42.159836 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.309997 (+150161us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.313592 (+ 3595us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.340118 (+ 26526us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7169609}
W20251026 02:56:42.344429 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a53aedec98894542aa3c419b2295c67d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:42.344936 2377 ts_tablet_manager.cc:1950] T 865fbd674db34f668fb5a2f5836a9b43 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:42.345443 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3b7d85a2941545ceaa16b22e8328b4ff (table default.Table_5ae1979aa25849fe8b780ecc617bb7e8 [id=9f6ac6400247491485d6bef30bf6cf4b]) successfully deleted
I20251026 02:56:42.340341 2375 raft_consensus.cc:359] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.345717 2375 raft_consensus.cc:385] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.345845 2375 raft_consensus.cc:740] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.346129 2375 consensus_queue.cc:260] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.347838 2375 ts_tablet_manager.cc:1434] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20251026 02:56:42.348505 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 690) took 6103 ms. Trace:
W20251026 02:56:42.349412 1629 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 252971us
I20251026 02:56:42.361891 2375 tablet_bootstrap.cc:492] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:42.363763 2375 tablet_bootstrap.cc:654] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:42.346661 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 680) took 7387 ms. Trace:
I20251026 02:56:42.364275 2372 rpcz_store.cc:276] 1026 02:56:34.959640 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.639715 (+680075us) service_pool.cc:225] Handling call
1026 02:56:42.346513 (+6706798us) ts_tablet_manager.cc:1151] Deleting tablet bc4ca4423e3a4e0c96846babec931fec
1026 02:56:42.346525 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.346640 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6706373}
W20251026 02:56:42.364894 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet bc4ca4423e3a4e0c96846babec931fec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc4ca4423e3a4e0c96846babec931fec already in progress: creating tablet
I20251026 02:56:42.365316 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 681) took 7405 ms. Trace:
I20251026 02:56:42.365414 2372 rpcz_store.cc:276] 1026 02:56:35.021353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.640159 (+618806us) service_pool.cc:225] Handling call
1026 02:56:42.365178 (+6725019us) ts_tablet_manager.cc:1151] Deleting tablet a74d6a4e041347578b0976a8b54bf22c
1026 02:56:42.365189 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.365297 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6724791}
W20251026 02:56:42.365833 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a74d6a4e041347578b0976a8b54bf22c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a74d6a4e041347578b0976a8b54bf22c already in progress: creating tablet
I20251026 02:56:42.366212 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 682) took 7204 ms. Trace:
I20251026 02:56:42.366291 2372 rpcz_store.cc:276] 1026 02:56:35.161597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.640398 (+478801us) service_pool.cc:225] Handling call
1026 02:56:42.366101 (+6725703us) ts_tablet_manager.cc:1151] Deleting tablet 657e07d0b17c44d4b78b97210f17ce40
1026 02:56:42.366111 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.366193 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6725441}
W20251026 02:56:42.366732 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 657e07d0b17c44d4b78b97210f17ce40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 657e07d0b17c44d4b78b97210f17ce40 already in progress: creating tablet
I20251026 02:56:42.367069 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 683) took 7152 ms. Trace:
I20251026 02:56:42.367151 2372 rpcz_store.cc:276] 1026 02:56:35.214366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.640683 (+426317us) service_pool.cc:225] Handling call
1026 02:56:42.366967 (+6726284us) ts_tablet_manager.cc:1151] Deleting tablet 3efafaac6c634d088396d56c8debd990
1026 02:56:42.366976 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.367051 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6726107}
W20251026 02:56:42.367560 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3efafaac6c634d088396d56c8debd990 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3efafaac6c634d088396d56c8debd990 already in progress: creating tablet
I20251026 02:56:42.367897 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 684) took 7056 ms. Trace:
I20251026 02:56:42.367976 2372 rpcz_store.cc:276] 1026 02:56:35.311036 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.640875 (+329839us) service_pool.cc:225] Handling call
1026 02:56:42.367797 (+6726922us) ts_tablet_manager.cc:1151] Deleting tablet 1219e8bdfde9444f9bfdf1d626389e88
1026 02:56:42.367806 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.367880 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6718197}
W20251026 02:56:42.368984 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 1219e8bdfde9444f9bfdf1d626389e88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1219e8bdfde9444f9bfdf1d626389e88 already in progress: creating tablet
I20251026 02:56:42.369407 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 693) took 6722 ms. Trace:
I20251026 02:56:42.369488 2372 rpcz_store.cc:276] 1026 02:56:35.647220 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.822416 (+175196us) service_pool.cc:225] Handling call
1026 02:56:42.369216 (+6546800us) ts_tablet_manager.cc:1151] Deleting tablet 1dbbfb77851d443f8cf4d798f3e07e59
1026 02:56:42.369289 (+ 73us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.369391 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6546594}
W20251026 02:56:42.369953 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 1dbbfb77851d443f8cf4d798f3e07e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dbbfb77851d443f8cf4d798f3e07e59 already in progress: creating tablet
I20251026 02:56:42.372283 2377 rpcz_store.cc:276] 1026 02:56:36.246042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.246130 (+ 88us) service_pool.cc:225] Handling call
1026 02:56:42.216810 (+5970680us) ts_tablet_manager.cc:1151] Deleting tablet 865fbd674db34f668fb5a2f5836a9b43
1026 02:56:42.216819 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.290552 (+ 73733us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.331284 (+ 40732us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.348484 (+ 17200us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5952781}
I20251026 02:56:42.372799 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 691) took 6115 ms. Trace:
I20251026 02:56:42.372915 2377 rpcz_store.cc:276] 1026 02:56:36.260142 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.260902 (+ 760us) service_pool.cc:225] Handling call
1026 02:56:42.372662 (+6111760us) ts_tablet_manager.cc:1151] Deleting tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3
1026 02:56:42.372674 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.372780 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6104229}
I20251026 02:56:42.373271 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 692) took 6111 ms. Trace:
I20251026 02:56:42.373381 2377 rpcz_store.cc:276] 1026 02:56:36.276563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.283153 (+ 6590us) service_pool.cc:225] Handling call
1026 02:56:42.373170 (+6090017us) ts_tablet_manager.cc:1151] Deleting tablet b8a1244c1c49471fb7711a2ca9b89a99
1026 02:56:42.373178 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.373256 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6089824}
I20251026 02:56:42.373823 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 702) took 5766 ms. Trace:
I20251026 02:56:42.373924 2377 rpcz_store.cc:276] 1026 02:56:36.607032 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.608065 (+ 1033us) service_pool.cc:225] Handling call
1026 02:56:42.373727 (+5765662us) ts_tablet_manager.cc:1151] Deleting tablet 1c740ab6481e4481b95878c3a185129d
1026 02:56:42.373735 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.373809 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5765508}
I20251026 02:56:42.374245 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 703) took 5767 ms. Trace:
I20251026 02:56:42.374342 2377 rpcz_store.cc:276] 1026 02:56:36.607105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.608273 (+ 1168us) service_pool.cc:225] Handling call
1026 02:56:42.374154 (+5765881us) ts_tablet_manager.cc:1151] Deleting tablet 64415f5588db4df0a94676b650fb3631
1026 02:56:42.374163 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.374232 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5765734}
I20251026 02:56:42.374662 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 705) took 5179 ms. Trace:
I20251026 02:56:42.374759 2377 rpcz_store.cc:276] 1026 02:56:37.194721 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.194794 (+ 73us) service_pool.cc:225] Handling call
1026 02:56:42.374568 (+5179774us) ts_tablet_manager.cc:1151] Deleting tablet c13d89fc829049428363ede24c350ee5
1026 02:56:42.374576 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.374646 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5179545}
I20251026 02:56:42.375082 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 706) took 5180 ms. Trace:
I20251026 02:56:42.375178 2377 rpcz_store.cc:276] 1026 02:56:37.194833 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.195076 (+ 243us) service_pool.cc:225] Handling call
1026 02:56:42.374991 (+5179915us) ts_tablet_manager.cc:1151] Deleting tablet f719c720cc474181a63786a3e248ab2c
1026 02:56:42.375000 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.375067 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5179764}
I20251026 02:56:42.375486 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 707) took 5180 ms. Trace:
I20251026 02:56:42.375574 2377 rpcz_store.cc:276] 1026 02:56:37.194913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.195128 (+ 215us) service_pool.cc:225] Handling call
1026 02:56:42.375416 (+5180288us) ts_tablet_manager.cc:1151] Deleting tablet 98fe44aab33b40eba819386c3bffca46
1026 02:56:42.375425 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.375473 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5180109}
I20251026 02:56:42.375891 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 708) took 5180 ms. Trace:
I20251026 02:56:42.391806 2376 ts_tablet_manager.cc:1929] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.395162 3778 raft_consensus.cc:493] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.395382 3778 raft_consensus.cc:515] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.395782 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 694) took 6728 ms. Trace:
I20251026 02:56:42.395876 2372 rpcz_store.cc:276] 1026 02:56:35.667417 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.822631 (+155214us) service_pool.cc:225] Handling call
1026 02:56:42.395624 (+6572993us) ts_tablet_manager.cc:1151] Deleting tablet 1ffead7ca2d946c78aa4b8a602865fc8
1026 02:56:42.395637 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.395757 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6572226}
I20251026 02:56:42.396135 3778 leader_election.cc:290] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:42.396476 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 1ffead7ca2d946c78aa4b8a602865fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ffead7ca2d946c78aa4b8a602865fc8 already in progress: creating tablet
I20251026 02:56:42.397557 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 695) took 6701 ms. Trace:
W20251026 02:56:42.340116 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 80972.9us
W20251026 02:56:42.386422 2194 consensus_peers.cc:597] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:42.398192 2195 leader_election.cc:304] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.400918 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 865fbd674db34f668fb5a2f5836a9b43 (table default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa]) successfully deleted
W20251026 02:56:42.401587 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 already in progress: creating tablet
W20251026 02:56:42.402016 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet b8a1244c1c49471fb7711a2ca9b89a99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8a1244c1c49471fb7711a2ca9b89a99 already in progress: creating tablet
I20251026 02:56:42.402148 3537 raft_consensus.cc:2804] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 02:56:42.402432 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 1c740ab6481e4481b95878c3a185129d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c740ab6481e4481b95878c3a185129d already in progress: creating tablet
W20251026 02:56:42.402839 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 64415f5588db4df0a94676b650fb3631 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64415f5588db4df0a94676b650fb3631 already in progress: creating tablet
I20251026 02:56:42.402853 3537 raft_consensus.cc:697] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
W20251026 02:56:42.403240 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet c13d89fc829049428363ede24c350ee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c13d89fc829049428363ede24c350ee5 already in progress: creating tablet
I20251026 02:56:42.403167 3537 consensus_queue.cc:237] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:42.403635 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet f719c720cc474181a63786a3e248ab2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f719c720cc474181a63786a3e248ab2c already in progress: creating tablet
W20251026 02:56:42.404017 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 98fe44aab33b40eba819386c3bffca46 because the tablet was not found. No further retry: Not found: Tablet not found: 98fe44aab33b40eba819386c3bffca46
I20251026 02:56:42.406421 2377 rpcz_store.cc:276] 1026 02:56:37.195459 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.195500 (+ 41us) service_pool.cc:225] Handling call
1026 02:56:42.375800 (+5180300us) ts_tablet_manager.cc:1151] Deleting tablet 58d60bf2a9e546e6a9474a5af6d5273a
1026 02:56:42.375807 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.375876 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5180164}
I20251026 02:56:42.399149 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c6e3e47b3e7421e945cfbe671b06abe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:42.399863 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c31d2704d1f14ddead69c794cc631ec1" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:42.406795 2109 raft_consensus.cc:3060] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:42.406903 2118 raft_consensus.cc:3060] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:42.409521 2109 raft_consensus.cc:2468] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:42.410030 2372 rpcz_store.cc:276] 1026 02:56:35.696453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.823422 (+126969us) service_pool.cc:225] Handling call
1026 02:56:42.397433 (+6574011us) ts_tablet_manager.cc:1151] Deleting tablet 24f16e40a30c41788a6f9813c235c5ab
1026 02:56:42.397443 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.397539 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6573817}
I20251026 02:56:42.410595 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 709) took 5214 ms. Trace:
I20251026 02:56:42.410307 2118 raft_consensus.cc:2468] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:42.411304 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 24f16e40a30c41788a6f9813c235c5ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24f16e40a30c41788a6f9813c235c5ab already in progress: creating tablet
I20251026 02:56:42.411763 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 696) took 6670 ms. Trace:
W20251026 02:56:42.412243 2194 consensus_peers.cc:597] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:42.412885 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c6a7c774044411a4733b115080c4c6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:42.412968 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a345fa5edbd48d2891b8b40b09cb3f8" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:42.416680 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 58d60bf2a9e546e6a9474a5af6d5273a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58d60bf2a9e546e6a9474a5af6d5273a already in progress: creating tablet
I20251026 02:56:42.417277 2377 rpcz_store.cc:276] 1026 02:56:37.195743 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.195779 (+ 36us) service_pool.cc:225] Handling call
1026 02:56:42.410441 (+5214662us) ts_tablet_manager.cc:1151] Deleting tablet 244c03a9293242a48077bdaf90a47865
1026 02:56:42.410452 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.410578 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5214486}
W20251026 02:56:42.418102 2195 consensus_peers.cc:597] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d65b0d79e2e54af9810022b683c3288b. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:42.418318 2195 consensus_peers.cc:597] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 741792cf96164695b03de3fb86b12602. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:42.420655 2195 leader_election.cc:304] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.423671 1662 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet de8799a5af2b42cc92ccc0195cff98e7 (table default.Table_13667e60be9c46b192cebf20df523e99 [id=7885bd43f8a245b0a3010a63dbd535f5]). Expected version 3 got 0
I20251026 02:56:42.426227 1661 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet ff242ef9541a42d793c4484190956da6 (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]). Expected version 1 got 0
I20251026 02:56:42.399734 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7177931353164b2aa82c00b2807ccd63" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:42.426476 1661 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet e9cd10092aac40068a1c7f85e514fd37 (table default.Table_fe09be2527a840b09e3e64c27965845d [id=de5272bcc29644208ddd95bdec21254e]). Expected version 1 got 0
I20251026 02:56:42.426576 2117 raft_consensus.cc:3060] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:42.426473 2372 rpcz_store.cc:276] 1026 02:56:35.741633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.823631 (+ 81998us) service_pool.cc:225] Handling call
1026 02:56:42.411638 (+6588007us) ts_tablet_manager.cc:1151] Deleting tablet 351cd60b076a49949f03640c981845c4
1026 02:56:42.411649 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.411746 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6587833}
I20251026 02:56:42.426739 1661 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 85f9fcae83a04c6e907c7e9c8cd95dbe (table default.Table_7244d7be0e764416aaba258478bdffdc [id=87349d9c6c764382b828e716e7854c59]). Expected version 4 got 0
W20251026 02:56:42.427066 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 351cd60b076a49949f03640c981845c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 351cd60b076a49949f03640c981845c4 already in progress: creating tablet
I20251026 02:56:42.425979 2375 tablet_bootstrap.cc:492] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:42.427549 2375 ts_tablet_manager.cc:1403] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.001s
I20251026 02:56:42.428455 2372 tablet_replica.cc:333] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:42.428669 2372 raft_consensus.cc:2243] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:42.428997 2372 raft_consensus.cc:2272] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.429394 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 710) took 5233 ms. Trace:
I20251026 02:56:42.429589 2117 raft_consensus.cc:2468] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:42.429787 2377 rpcz_store.cc:276] 1026 02:56:37.196090 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.196139 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:42.417724 (+5221585us) ts_tablet_manager.cc:1151] Deleting tablet ded06ba1f80c4433ae78687e5990edce
1026 02:56:42.417735 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.429368 (+ 11633us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5221349}
I20251026 02:56:42.431681 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 713) took 5155 ms. Trace:
I20251026 02:56:42.432149 2375 raft_consensus.cc:359] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.433501 2377 rpcz_store.cc:276] 1026 02:56:37.275928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.278985 (+ 3057us) service_pool.cc:225] Handling call
1026 02:56:42.430904 (+5151919us) ts_tablet_manager.cc:1151] Deleting tablet 1a36b8a4beca483bb853d27961f5281a
1026 02:56:42.431528 (+ 624us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.431661 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5151628}
I20251026 02:56:42.434211 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 714) took 5158 ms. Trace:
I20251026 02:56:42.434448 2375 raft_consensus.cc:385] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.434669 2375 raft_consensus.cc:740] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.436144 2377 rpcz_store.cc:276] 1026 02:56:37.276050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.279326 (+ 3276us) service_pool.cc:225] Handling call
1026 02:56:42.434069 (+5154743us) ts_tablet_manager.cc:1151] Deleting tablet 7e1a45412db1449a99252112d8110a17
1026 02:56:42.434078 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.434192 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5154583}
I20251026 02:56:42.436594 2375 consensus_queue.cc:260] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.436944 2376 log.cc:1199] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/aa0aa6cdd0de4b0a91f553c2341e9c01
I20251026 02:56:42.437299 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 715) took 5161 ms. Trace:
I20251026 02:56:42.433418 2372 ts_tablet_manager.cc:1916] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.441486 2376 ts_tablet_manager.cc:1950] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:42.442075 2375 ts_tablet_manager.cc:1434] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251026 02:56:42.442235 2377 rpcz_store.cc:276] 1026 02:56:37.276157 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.279524 (+ 3367us) service_pool.cc:225] Handling call
1026 02:56:42.437153 (+5157629us) ts_tablet_manager.cc:1151] Deleting tablet b68388b2fd014dd1a5f6572796cf53a6
1026 02:56:42.437163 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.437281 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5157464}
I20251026 02:56:42.442739 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 716) took 5156 ms. Trace:
I20251026 02:56:42.443663 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 715) took 4398 ms. Trace:
I20251026 02:56:42.444047 2375 tablet_bootstrap.cc:492] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:42.445803 2375 tablet_bootstrap.cc:654] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:42.444159 2377 rpcz_store.cc:276] 1026 02:56:37.288703 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.288779 (+ 76us) service_pool.cc:225] Handling call
1026 02:56:42.442607 (+5153828us) ts_tablet_manager.cc:1151] Deleting tablet 81775055d98349f696cdab6171ffd58b
1026 02:56:42.442618 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.442721 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5153585}
I20251026 02:56:42.446412 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 717) took 5157 ms. Trace:
I20251026 02:56:42.446543 2377 rpcz_store.cc:276] 1026 02:56:37.288850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.289081 (+ 231us) service_pool.cc:225] Handling call
1026 02:56:42.446283 (+5157202us) ts_tablet_manager.cc:1151] Deleting tablet e830f1aa1270420c9b28a0491a1a0960
1026 02:56:42.446293 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.446395 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5157038}
I20251026 02:56:42.446776 2376 rpcz_store.cc:276] 1026 02:56:38.045453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.048301 (+ 2848us) service_pool.cc:225] Handling call
1026 02:56:42.333829 (+4285528us) ts_tablet_manager.cc:1151] Deleting tablet aa0aa6cdd0de4b0a91f553c2341e9c01
1026 02:56:42.333840 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.388435 (+ 54595us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.391690 (+ 3255us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.443638 (+ 51948us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4285254}
I20251026 02:56:42.447238 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 718) took 5158 ms. Trace:
I20251026 02:56:42.447873 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet aa0aa6cdd0de4b0a91f553c2341e9c01 (table default.Table_1a48da091e984e09aaffb9caed79f2c1 [id=1b3988cec795459f8666476a9f979f3a]) successfully deleted
I20251026 02:56:42.448369 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 716) took 4241 ms. Trace:
I20251026 02:56:42.452247 2377 rpcz_store.cc:276] 1026 02:56:37.288964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.292685 (+ 3721us) service_pool.cc:225] Handling call
1026 02:56:42.447125 (+5154440us) ts_tablet_manager.cc:1151] Deleting tablet e32a1f8dcad4430eb35d388bfad7d110
1026 02:56:42.447134 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.447222 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5154232}
I20251026 02:56:42.452695 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 726) took 4702 ms. Trace:
I20251026 02:56:42.452847 2376 rpcz_store.cc:276] 1026 02:56:38.206974 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.207027 (+ 53us) service_pool.cc:225] Handling call
1026 02:56:42.448236 (+4241209us) ts_tablet_manager.cc:1151] Deleting tablet fe48472efa444bf580e4074c96a9de92
1026 02:56:42.448247 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.448347 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4240977}
W20251026 02:56:42.453642 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet fe48472efa444bf580e4074c96a9de92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe48472efa444bf580e4074c96a9de92 already in progress: creating tablet
I20251026 02:56:42.453923 2377 rpcz_store.cc:276] 1026 02:56:37.750603 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.750684 (+ 81us) service_pool.cc:225] Handling call
1026 02:56:42.452584 (+4701900us) ts_tablet_manager.cc:1151] Deleting tablet ba08469339184b11b9ced9e6bdd63d42
1026 02:56:42.452595 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.452679 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4701681}
I20251026 02:56:42.454334 2376 tablet_replica.cc:333] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:42.457262 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 728) took 4706 ms. Trace:
I20251026 02:56:42.458626 2376 raft_consensus.cc:2243] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.458782 2377 rpcz_store.cc:276] 1026 02:56:37.750761 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.751133 (+ 372us) service_pool.cc:225] Handling call
1026 02:56:42.457170 (+4706037us) ts_tablet_manager.cc:1151] Deleting tablet 22e4a433801a4cb58062d3e76c760b69
1026 02:56:42.457181 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.457246 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4705866}
I20251026 02:56:42.459192 2376 raft_consensus.cc:2272] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.459492 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 730) took 4708 ms. Trace:
I20251026 02:56:42.460166 2376 ts_tablet_manager.cc:1916] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.460314 2377 rpcz_store.cc:276] 1026 02:56:37.751126 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.751483 (+ 357us) service_pool.cc:225] Handling call
1026 02:56:42.459403 (+4707920us) ts_tablet_manager.cc:1151] Deleting tablet a111848706b743c9beb2fbd21f59d7f3
1026 02:56:42.459413 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.459478 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4707752}
I20251026 02:56:42.460685 2377 tablet_replica.cc:333] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:42.461835 2377 raft_consensus.cc:2243] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:42.430408 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 6108219254bd448c852096c0c967384b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.430483 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a3491b303f94fe2a2c838297f73d464 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b5e61afe90e4bc4a2897de21e0b3919) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.398250 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a345fa5edbd48d2891b8b40b09cb3f8" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:42.462604 2053 leader_election.cc:343] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.463783 2165 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:42.468688 3239 raft_consensus.cc:2804] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:42.468868 3239 raft_consensus.cc:697] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:42.469163 3239 consensus_queue.cc:237] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.471529 3750 tablet.cc:1722] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:42.471779 2377 raft_consensus.cc:2272] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.482547 3750 tablet.cc:1722] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:42.486591 2375 tablet_bootstrap.cc:492] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:42.486665 3372 raft_consensus.cc:493] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.486819 2375 ts_tablet_manager.cc:1403] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20251026 02:56:42.486843 3372 raft_consensus.cc:515] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.487536 3372 leader_election.cc:290] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:42.487643 2375 raft_consensus.cc:359] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.487854 2375 raft_consensus.cc:385] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.487902 2375 raft_consensus.cc:740] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.490336 2377 ts_tablet_manager.cc:1916] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:42.495546 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 74920.6us
I20251026 02:56:42.495890 2195 leader_election.cc:304] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.495043 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff242ef9541a42d793c4484190956da6" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:42.502259 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 3513c740306a4c4c97a4e33849775fdc (DEFAULT_TABLE table=default.Table_2811eabdf1ec4f61b92ab33a8c7ec6c3 [id=d221b3561025478aa3ba2410ee56094f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.502708 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3513c740306a4c4c97a4e33849775fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.503850 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 3513c740306a4c4c97a4e33849775fdc (DEFAULT_TABLE table=default.Table_2811eabdf1ec4f61b92ab33a8c7ec6c3 [id=d221b3561025478aa3ba2410ee56094f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.504349 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3513c740306a4c4c97a4e33849775fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.514283 3616 raft_consensus.cc:2804] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:42.514603 3616 raft_consensus.cc:697] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:42.514990 3616 consensus_queue.cc:237] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.517702 2372 ts_tablet_manager.cc:1929] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.517897 2372 log.cc:1199] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/a6f9ae327aa4401baf66a737eaa93cf6
I20251026 02:56:42.518359 2372 ts_tablet_manager.cc:1950] T a6f9ae327aa4401baf66a737eaa93cf6 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:42.518721 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 697) took 6691 ms. Trace:
I20251026 02:56:42.518872 2372 rpcz_store.cc:276] 1026 02:56:35.827112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.830272 (+ 3160us) service_pool.cc:225] Handling call
1026 02:56:42.428392 (+6598120us) ts_tablet_manager.cc:1151] Deleting tablet a6f9ae327aa4401baf66a737eaa93cf6
1026 02:56:42.428404 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.460801 (+ 32397us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.517583 (+ 56782us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.518706 (+ 1123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6590866}
I20251026 02:56:42.519501 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet a6f9ae327aa4401baf66a737eaa93cf6 (table default.Table_828c16a96a47489f835a891f34d707b1 [id=432234d273ca4c9ea53c7a078c274637]) successfully deleted
I20251026 02:56:42.519969 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 690) took 7023 ms. Trace:
I20251026 02:56:42.520613 2372 rpcz_store.cc:276] 1026 02:56:35.496599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.819366 (+322767us) service_pool.cc:225] Handling call
1026 02:56:42.519843 (+6700477us) ts_tablet_manager.cc:1151] Deleting tablet 902cc4f9dbbd4d8baa829448f2a4687a
1026 02:56:42.519853 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.519954 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6666987}
I20251026 02:56:42.521544 3815 tablet.cc:1722] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251026 02:56:42.521611 3537 raft_consensus.cc:493] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.522050 3537 raft_consensus.cc:515] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:42.524000 3537 leader_election.cc:290] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.524431 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbf2a1ce78154415b23ab6d0efbc1f52" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:42.524681 1980 raft_consensus.cc:2468] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:42.525375 2194 leader_election.cc:304] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:42.525621 3517 raft_consensus.cc:2804] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 02:56:42.525693 3517 raft_consensus.cc:493] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:42.525741 3517 raft_consensus.cc:3060] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:42.527427 1980 raft_consensus.cc:1275] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 02:56:42.527983 3622 consensus_queue.cc:1048] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:42.528319 3517 raft_consensus.cc:515] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:42.529202 3517 leader_election.cc:290] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.529433 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbf2a1ce78154415b23ab6d0efbc1f52" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:42.529640 1978 raft_consensus.cc:3060] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:42.533075 1978 raft_consensus.cc:2468] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 2.
I20251026 02:56:42.534791 2194 leader_election.cc:304] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:42.535116 3537 raft_consensus.cc:2804] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 02:56:42.535492 3537 raft_consensus.cc:697] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 2 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:42.536181 3537 consensus_queue.cc:237] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:42.536485 2195 leader_election.cc:304] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:42.537339 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 902cc4f9dbbd4d8baa829448f2a4687a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 902cc4f9dbbd4d8baa829448f2a4687a already in progress: creating tablet
I20251026 02:56:42.538566 2372 tablet_replica.cc:333] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:42.538813 2372 raft_consensus.cc:2243] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.539005 2372 raft_consensus.cc:2272] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.539599 2372 ts_tablet_manager.cc:1916] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.539551 3816 tablet.cc:1722] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:42.520278 2375 consensus_queue.cc:260] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.541739 2375 ts_tablet_manager.cc:1434] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.055s user 0.002s sys 0.000s
I20251026 02:56:42.542068 2375 tablet_bootstrap.cc:492] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:42.543915 2375 tablet_bootstrap.cc:654] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:42.544086 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_37167b3e100a458d81c0578a964bee27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:42.544778 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37167b3e100a458d81c0578a964bee27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:42.545518 3622 raft_consensus.cc:2804] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:42.545670 3622 raft_consensus.cc:697] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:42.545933 3622 consensus_queue.cc:237] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.547498 2377 ts_tablet_manager.cc:1929] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.548444 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 5b8e1d79d2a247dba25540daf6b81bf0
I20251026 02:56:42.552304 2377 log.cc:1199] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/1d7ffdaeec2240dcbe4cafdb96073a7d
I20251026 02:56:42.553635 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b8e1d79d2a247dba25540daf6b81bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:42.554167 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 2f197526e7d34f849dbb4f28d8d414cf
I20251026 02:56:42.555444 2377 ts_tablet_manager.cc:1950] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:42.556272 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 732) took 4804 ms. Trace:
W20251026 02:56:42.556377 2053 leader_election.cc:343] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.556349 2377 rpcz_store.cc:276] 1026 02:56:37.751450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.751494 (+ 44us) service_pool.cc:225] Handling call
1026 02:56:42.460636 (+4709142us) ts_tablet_manager.cc:1151] Deleting tablet 1d7ffdaeec2240dcbe4cafdb96073a7d
1026 02:56:42.460647 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.493436 (+ 32789us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.547388 (+ 53952us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.556251 (+ 8863us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3244,"tablet-delete.queue_time_us":4708882}
I20251026 02:56:42.556535 2053 leader_election.cc:304] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.556654 2377 tablet_replica.cc:333] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:42.556824 2377 raft_consensus.cc:2243] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:42.557150 2377 raft_consensus.cc:2272] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.557410 3778 raft_consensus.cc:2749] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:42.561470 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f197526e7d34f849dbb4f28d8d414cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:42.564708 2375 tablet_bootstrap.cc:492] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:42.564951 2375 ts_tablet_manager.cc:1403] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251026 02:56:42.565780 2375 raft_consensus.cc:359] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.566023 2375 raft_consensus.cc:385] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251026 02:56:42.566044 1647 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 (table default.Table_1821574ebc7f452b9cc0d8231f78cbdc [id=216da41d45b24046b44678c1ff34c17e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 consensus state has no leader
I20251026 02:56:42.566130 2375 raft_consensus.cc:740] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.566368 2375 consensus_queue.cc:260] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.567059 2375 ts_tablet_manager.cc:1434] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:42.567407 2375 tablet_bootstrap.cc:492] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:42.569695 2377 ts_tablet_manager.cc:1916] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.546135 2376 ts_tablet_manager.cc:1929] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.555071 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b8e1d79d2a247dba25540daf6b81bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.557616 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f197526e7d34f849dbb4f28d8d414cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.571506 2376 log.cc:1199] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/d90e62ae86b149b291dc29cf7442c6bc
I20251026 02:56:42.571988 2376 ts_tablet_manager.cc:1950] T d90e62ae86b149b291dc29cf7442c6bc P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:42.572438 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 717) took 4301 ms. Trace:
I20251026 02:56:42.572589 2376 rpcz_store.cc:276] 1026 02:56:38.270588 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.272613 (+ 2025us) service_pool.cc:225] Handling call
1026 02:56:42.454248 (+4181635us) ts_tablet_manager.cc:1151] Deleting tablet d90e62ae86b149b291dc29cf7442c6bc
1026 02:56:42.454260 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.465985 (+ 11725us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.545998 (+ 80013us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.572399 (+ 26401us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4181333}
I20251026 02:56:42.573189 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet d90e62ae86b149b291dc29cf7442c6bc (table default.Table_e2a5e05b70f841cfb0065041b39e89cb [id=e1c065cb2f704594873006d80a406df2]) successfully deleted
I20251026 02:56:42.574090 2376 tablet_replica.cc:333] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:42.574371 2376 raft_consensus.cc:2243] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.574642 2376 raft_consensus.cc:2272] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.575222 1945 tablet_service.cc:1505] Processing CreateTablet for tablet aa1253c60ffb4b37849c9318307b94f1 (DEFAULT_TABLE table=default.Table_a636a385c0844582a777f2be1f382310 [id=ae5163beeb224e3f8e26cf4dffbbb231]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.576179 2376 ts_tablet_manager.cc:1916] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.569088 2375 tablet_bootstrap.cc:654] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:42.579595 2053 leader_election.cc:343] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.579790 2053 leader_election.cc:304] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.580356 3778 raft_consensus.cc:2749] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:42.584594 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 244c03a9293242a48077bdaf90a47865 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 244c03a9293242a48077bdaf90a47865 already in progress: creating tablet
W20251026 02:56:42.584864 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ded06ba1f80c4433ae78687e5990edce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ded06ba1f80c4433ae78687e5990edce already in progress: creating tablet
W20251026 02:56:42.585088 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 1a36b8a4beca483bb853d27961f5281a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a36b8a4beca483bb853d27961f5281a already in progress: creating tablet
W20251026 02:56:42.585273 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7e1a45412db1449a99252112d8110a17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1a45412db1449a99252112d8110a17 already in progress: creating tablet
W20251026 02:56:42.585471 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet b68388b2fd014dd1a5f6572796cf53a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b68388b2fd014dd1a5f6572796cf53a6 already in progress: creating tablet
W20251026 02:56:42.585654 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 81775055d98349f696cdab6171ffd58b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81775055d98349f696cdab6171ffd58b already in progress: creating tablet
W20251026 02:56:42.585966 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet e32a1f8dcad4430eb35d388bfad7d110 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e32a1f8dcad4430eb35d388bfad7d110 already in progress: creating tablet
W20251026 02:56:42.586113 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ba08469339184b11b9ced9e6bdd63d42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba08469339184b11b9ced9e6bdd63d42 already in progress: creating tablet
W20251026 02:56:42.586315 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 22e4a433801a4cb58062d3e76c760b69 because the tablet was not found. No further retry: Not found: Tablet not found: 22e4a433801a4cb58062d3e76c760b69
I20251026 02:56:42.591866 2096 tablet_service.cc:1505] Processing CreateTablet for tablet aa1253c60ffb4b37849c9318307b94f1 (DEFAULT_TABLE table=default.Table_a636a385c0844582a777f2be1f382310 [id=ae5163beeb224e3f8e26cf4dffbbb231]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.592661 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa1253c60ffb4b37849c9318307b94f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.593833 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa1253c60ffb4b37849c9318307b94f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.598920 2381 tablet_bootstrap.cc:492] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:42.599125 2381 ts_tablet_manager.cc:1403] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.493s user 0.038s sys 0.148s
I20251026 02:56:42.599954 2381 raft_consensus.cc:359] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.603415 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 663076e362c44837beb322aa965cde42
I20251026 02:56:42.603992 1655 catalog_manager.cc:6719] ReplaceTablet: tablet 663076e362c44837beb322aa965cde42 deleted and replaced by tablet 56d7760f605944c2886d0471c190d3af
I20251026 02:56:42.604290 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:42.605301 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8977f824ff374d05b4ed9d88c67329f9
I20251026 02:56:42.605435 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 663076e362c44837beb322aa965cde42 with delete_type TABLET_DATA_DELETED (replaced by tablet 56d7760f605944c2886d0471c190d3af) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:42.606766 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 8977f824ff374d05b4ed9d88c67329f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:42.608800 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 8977f824ff374d05b4ed9d88c67329f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.610386 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a4f4d7f87916425999574fd5b9f2fdf7
I20251026 02:56:42.610895 1663 catalog_manager.cc:6719] ReplaceTablet: tablet a4f4d7f87916425999574fd5b9f2fdf7 deleted and replaced by tablet ab923e219ff442939b66f1ad0902151c
I20251026 02:56:42.611681 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet a4f4d7f87916425999574fd5b9f2fdf7 with delete_type TABLET_DATA_DELETED (replaced by tablet ab923e219ff442939b66f1ad0902151c) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:42.612937 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet a4f4d7f87916425999574fd5b9f2fdf7 with delete_type TABLET_DATA_DELETED (replaced by tablet ab923e219ff442939b66f1ad0902151c) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.613790 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ac82c14e7dcb448b845d998249e2b339
I20251026 02:56:42.615027 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet ac82c14e7dcb448b845d998249e2b339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:42.617440 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet ac82c14e7dcb448b845d998249e2b339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.618180 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2d20995352a745b1a63b0cb745126074"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:42.618958 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d20995352a745b1a63b0cb745126074 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:42.620486 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e0971fd722424b9cb2a4067114d03cfe" } new_table_name: "default.Table_08fb40faf4cf4203a28cfb8acb4276a8" modify_external_catalogs: true
I20251026 02:56:42.620682 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 663076e362c44837beb322aa965cde42 with delete_type TABLET_DATA_DELETED (replaced by tablet 56d7760f605944c2886d0471c190d3af) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.621712 2375 tablet_bootstrap.cc:492] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:42.621901 2375 ts_tablet_manager.cc:1403] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20251026 02:56:42.622769 2375 raft_consensus.cc:359] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.623026 2375 raft_consensus.cc:385] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.623143 2375 raft_consensus.cc:740] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.624060 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b8fe7825fa8d46669772cf84bc4b3afc
I20251026 02:56:42.625731 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet b8fe7825fa8d46669772cf84bc4b3afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:42.625816 2372 ts_tablet_manager.cc:1929] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.626056 2372 log.cc:1199] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/6a3491b303f94fe2a2c838297f73d464
I20251026 02:56:42.626531 2372 ts_tablet_manager.cc:1950] T 6a3491b303f94fe2a2c838297f73d464 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:42.626832 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet b8fe7825fa8d46669772cf84bc4b3afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:42.627455 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 7e326907a7b444969c82df0e066be160 (DEFAULT_TABLE table=default.Table_df93f777839e490e8edbf5380bf106ee [id=367b18a1ad094dd8a257ff062cd49dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.627878 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e326907a7b444969c82df0e066be160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.630898 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 4e191bb4035c46f0870b1d6ea80c92bb (DEFAULT_TABLE table=default.Table_b6bd6467e2f24020b6204f82698a81ce [id=16b3cc23358d4ce08aa68445e59b05ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.631295 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e191bb4035c46f0870b1d6ea80c92bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.631489 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 7e326907a7b444969c82df0e066be160 (DEFAULT_TABLE table=default.Table_df93f777839e490e8edbf5380bf106ee [id=367b18a1ad094dd8a257ff062cd49dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.631947 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e326907a7b444969c82df0e066be160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.635564 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 4e191bb4035c46f0870b1d6ea80c92bb (DEFAULT_TABLE table=default.Table_b6bd6467e2f24020b6204f82698a81ce [id=16b3cc23358d4ce08aa68445e59b05ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.635951 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e191bb4035c46f0870b1d6ea80c92bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.639060 1657 catalog_manager.cc:5649] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:42.639554 1657 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet de8799a5af2b42cc92ccc0195cff98e7 (table default.Table_13667e60be9c46b192cebf20df523e99 [id=7885bd43f8a245b0a3010a63dbd535f5]). Expected version 3 got 0
I20251026 02:56:42.640326 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_df944b7ec6554ea687b16b286ba44c55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:42.641010 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df944b7ec6554ea687b16b286ba44c55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:42.642695 1657 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet ff242ef9541a42d793c4484190956da6 (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]). Expected version 1 got 0
I20251026 02:56:42.642731 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_34885bd9ba024c96a4bea0c2fedcfb4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:42.643301 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34885bd9ba024c96a4bea0c2fedcfb4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:42.645345 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_797feb703f214aac9ea3dccac59e202a" } new_table_name: "default.Table_7eb7fd8131cc4bce81b6cc09c5062fc0" modify_external_catalogs: true
I20251026 02:56:42.645323 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_30a03eb811714167a1169b464b284eef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:42.645865 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30a03eb811714167a1169b464b284eef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:42.647591 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6f950e766d4c46d4be2009b0c46bcfb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:42.648239 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f950e766d4c46d4be2009b0c46bcfb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:42.651712 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_bc0096c9da9c4603a0d2ea3538682695" } new_table_name: "default.Table_fc6775059a3d41bdac1af3af53ffd921" modify_external_catalogs: true
I20251026 02:56:42.623551 2375 consensus_queue.cc:260] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.656013 2375 ts_tablet_manager.cc:1434] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20251026 02:56:42.659093 3778 raft_consensus.cc:493] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.659271 3778 raft_consensus.cc:515] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.659870 3778 leader_election.cc:290] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:42.662586 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fb2ff87cf0248a0a4a77f2733ef867b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:42.663378 2053 leader_election.cc:343] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.663456 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7941853c36584b5591a75bf490bdb543"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:42.666541 2378 tablet_bootstrap.cc:492] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:42.666742 2378 ts_tablet_manager.cc:1403] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.493s user 0.036s sys 0.114s
I20251026 02:56:42.667619 2378 raft_consensus.cc:359] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.667898 2378 raft_consensus.cc:385] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.668002 2378 raft_consensus.cc:740] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.668254 2378 consensus_queue.cc:260] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.668972 2378 ts_tablet_manager.cc:1434] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:42.669292 2378 tablet_bootstrap.cc:492] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:42.664899 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 3513c740306a4c4c97a4e33849775fdc (DEFAULT_TABLE table=default.Table_2811eabdf1ec4f61b92ab33a8c7ec6c3 [id=d221b3561025478aa3ba2410ee56094f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.669406 2381 raft_consensus.cc:385] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.665410 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b8e1d79d2a247dba25540daf6b81bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:42.669481 2381 raft_consensus.cc:740] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.669719 2381 consensus_queue.cc:260] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.665562 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 663076e362c44837beb322aa965cde42 with delete_type TABLET_DATA_DELETED (replaced by tablet 56d7760f605944c2886d0471c190d3af) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:42.665562 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f197526e7d34f849dbb4f28d8d414cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:42.665659 2228 tablet_service.cc:1505] Processing CreateTablet for tablet aa1253c60ffb4b37849c9318307b94f1 (DEFAULT_TABLE table=default.Table_a636a385c0844582a777f2be1f382310 [id=ae5163beeb224e3f8e26cf4dffbbb231]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.665830 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 8977f824ff374d05b4ed9d88c67329f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:42.670322 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3513c740306a4c4c97a4e33849775fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.666082 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet a4f4d7f87916425999574fd5b9f2fdf7 with delete_type TABLET_DATA_DELETED (replaced by tablet ab923e219ff442939b66f1ad0902151c) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:42.666191 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet ac82c14e7dcb448b845d998249e2b339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:42.666312 2227 tablet_service.cc:1552] Processing DeleteTablet for tablet b8fe7825fa8d46669772cf84bc4b3afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:42.666528 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 7e326907a7b444969c82df0e066be160 (DEFAULT_TABLE table=default.Table_df93f777839e490e8edbf5380bf106ee [id=367b18a1ad094dd8a257ff062cd49dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.666957 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 4e191bb4035c46f0870b1d6ea80c92bb (DEFAULT_TABLE table=default.Table_b6bd6467e2f24020b6204f82698a81ce [id=16b3cc23358d4ce08aa68445e59b05ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.667309 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff242ef9541a42d793c4484190956da6" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:42.671573 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa1253c60ffb4b37849c9318307b94f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.672433 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e326907a7b444969c82df0e066be160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.672497 2377 ts_tablet_manager.cc:1929] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.672573 2377 log.cc:1199] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/ce922040298d4952af13652eeda7a234
I20251026 02:56:42.672925 2377 ts_tablet_manager.cc:1950] T ce922040298d4952af13652eeda7a234 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:42.673182 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 731) took 4921 ms. Trace:
I20251026 02:56:42.673254 2377 rpcz_store.cc:276] 1026 02:56:37.751358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.751796 (+ 438us) service_pool.cc:225] Handling call
1026 02:56:42.556604 (+4804808us) ts_tablet_manager.cc:1151] Deleting tablet ce922040298d4952af13652eeda7a234
1026 02:56:42.556614 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.629020 (+ 72406us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.672403 (+ 43383us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.673168 (+ 765us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11902,"tablet-delete.queue_time_us":4804670}
I20251026 02:56:42.673579 2377 tablet_replica.cc:333] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:42.673746 2377 raft_consensus.cc:2243] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 2 LEADER]: Raft consensus shutting down.
I20251026 02:56:42.673928 2377 pending_rounds.cc:70] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:42.684659 2375 tablet_bootstrap.cc:492] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:42.686524 2375 tablet_bootstrap.cc:654] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:42.687364 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 698) took 6857 ms. Trace:
I20251026 02:56:42.688649 2372 rpcz_store.cc:276] 1026 02:56:35.844292 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:35.852336 (+ 8044us) service_pool.cc:225] Handling call
1026 02:56:42.538506 (+6686170us) ts_tablet_manager.cc:1151] Deleting tablet 6a3491b303f94fe2a2c838297f73d464
1026 02:56:42.538519 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.561500 (+ 22981us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.625677 (+ 64177us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.687321 (+ 61644us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6655337}
I20251026 02:56:42.689111 2372 tablet_replica.cc:333] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:42.689374 2372 raft_consensus.cc:2243] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.689616 2372 raft_consensus.cc:2272] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.690202 2372 ts_tablet_manager.cc:1916] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.676457 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e191bb4035c46f0870b1d6ea80c92bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.692270 2377 pending_rounds.cc:77] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7214888557558775808 op_type: NO_OP noop_request { }
I20251026 02:56:42.692487 2377 raft_consensus.cc:2889] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:42.692638 2377 raft_consensus.cc:2272] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.693519 2377 ts_tablet_manager.cc:1916] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.680563 2381 ts_tablet_manager.cc:1434] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.081s user 0.001s sys 0.001s
I20251026 02:56:42.701972 2381 tablet_bootstrap.cc:492] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:42.704046 2381 tablet_bootstrap.cc:654] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:42.714725 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 45518.9us
W20251026 02:56:42.715958 2195 leader_election.cc:343] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: dbf2a1ce78154415b23ab6d0efbc1f52
W20251026 02:56:42.717532 2195 leader_election.cc:343] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: dbf2a1ce78154415b23ab6d0efbc1f52
I20251026 02:56:42.717876 3769 raft_consensus.cc:493] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.718039 3769 raft_consensus.cc:515] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.719403 3769 leader_election.cc:290] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.722357 2375 tablet_bootstrap.cc:492] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:42.722626 2375 ts_tablet_manager.cc:1403] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.005s
I20251026 02:56:42.723735 2375 raft_consensus.cc:359] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.723965 2375 raft_consensus.cc:385] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.724028 2375 raft_consensus.cc:740] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.724366 2375 consensus_queue.cc:260] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.725029 2375 ts_tablet_manager.cc:1434] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:42.725420 2375 tablet_bootstrap.cc:492] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:42.725432 2377 ts_tablet_manager.cc:1929] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251026 02:56:42.682935 1975 raft_consensus.cc:3060] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:42.719385 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d212170f056a44dfb15cfcb7f9fbae2f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:42.727800 2377 log.cc:1199] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/47037db3602246a2ab6224557286162f
W20251026 02:56:42.728000 2194 leader_election.cc:343] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.728225 2377 ts_tablet_manager.cc:1950] T 47037db3602246a2ab6224557286162f P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:42.728586 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 736) took 4976 ms. Trace:
I20251026 02:56:42.728708 2377 rpcz_store.cc:276] 1026 02:56:37.752159 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.752202 (+ 43us) service_pool.cc:225] Handling call
1026 02:56:42.673539 (+4921337us) ts_tablet_manager.cc:1151] Deleting tablet 47037db3602246a2ab6224557286162f
1026 02:56:42.673546 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.697779 (+ 24233us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.725330 (+ 27551us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.728570 (+ 3240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4921177}
I20251026 02:56:42.729105 2377 tablet_replica.cc:333] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:42.729333 2377 raft_consensus.cc:2243] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:42.729655 2377 raft_consensus.cc:2272] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.730947 2377 ts_tablet_manager.cc:1916] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.733239 2378 tablet_bootstrap.cc:654] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:42.741823 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fb2ff87cf0248a0a4a77f2733ef867b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:42.720973 2376 ts_tablet_manager.cc:1929] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.742341 2376 log.cc:1199] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/2524ee0c964345459c3e62f1e39b8f9e
W20251026 02:56:42.744091 2053 leader_election.cc:343] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.744321 2053 leader_election.cc:304] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.744866 3778 raft_consensus.cc:2749] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:42.745404 3778 consensus_queue.cc:1048] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:42.746187 3830 tablet.cc:1722] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:42.746460 3778 consensus_queue.cc:1048] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:42.742736 1915 leader_election.cc:304] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:42.754323 3372 raft_consensus.cc:2749] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:42.755064 2376 ts_tablet_manager.cc:1950] T 2524ee0c964345459c3e62f1e39b8f9e P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:42.755882 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 719) took 4265 ms. Trace:
I20251026 02:56:42.756402 2376 rpcz_store.cc:276] 1026 02:56:38.490730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.490780 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:42.574013 (+4083233us) ts_tablet_manager.cc:1151] Deleting tablet 2524ee0c964345459c3e62f1e39b8f9e
1026 02:56:42.574025 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.717653 (+143628us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.720882 (+ 3229us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.755864 (+ 34982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4083041}
I20251026 02:56:42.757301 2376 tablet_replica.cc:333] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
W20251026 02:56:42.759328 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7941853c36584b5591a75bf490bdb543 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:42.760355 2376 raft_consensus.cc:2243] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.761984 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 6a3491b303f94fe2a2c838297f73d464 (table default.Table_30b96d2d4369458390803c6be18efe0e [id=9fd6e480fc3543149959773e691a3ddd]) successfully deleted
I20251026 02:56:42.761770 3835 tablet.cc:1722] T 85f9fcae83a04c6e907c7e9c8cd95dbe P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20251026 02:56:42.762812 1648 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 69549.3us
W20251026 02:56:42.763332 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 899a951f4f5d429b9a2a41cea2542a45 consensus state has no leader
I20251026 02:56:42.769282 2096 tablet_service.cc:1505] Processing CreateTablet for tablet c15e3cec8df74a56adfa7a5d5c1d23a2 (DEFAULT_TABLE table=default.Table_7175e585b5e549dc89c9f1eebaae5e96 [id=380e7a7694ec4b019f13d7a649697225]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.769781 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c15e3cec8df74a56adfa7a5d5c1d23a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.772269 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9dc1a989050c4cf48b9a8111a722b2ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:42.772868 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dc1a989050c4cf48b9a8111a722b2ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:42.774000 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 1d7ffdaeec2240dcbe4cafdb96073a7d (table default.Table_7b08384413864da0a28224dd7201c263 [id=a9255e57e54144ba876815cbb6155360]) successfully deleted
W20251026 02:56:42.774189 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 106790us
I20251026 02:56:42.775169 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 2524ee0c964345459c3e62f1e39b8f9e (table default.Table_164cb1784f4645e499339f7163fdfc2b [id=91b4b1a8162441299c680e7c202a96d9]) successfully deleted
I20251026 02:56:42.775462 3622 raft_consensus.cc:493] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:42.775611 3622 raft_consensus.cc:515] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:42.775781 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 47037db3602246a2ab6224557286162f (table default.Table_09ca38422af543f89eab26b4117536dc [id=12a8deac66914c6da970cd7f975b5b59]) successfully deleted
I20251026 02:56:42.776232 3622 leader_election.cc:290] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.776345 3622 raft_consensus.cc:493] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:42.776439 3622 raft_consensus.cc:515] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.776475 1945 tablet_service.cc:1505] Processing CreateTablet for tablet c15e3cec8df74a56adfa7a5d5c1d23a2 (DEFAULT_TABLE table=default.Table_7175e585b5e549dc89c9f1eebaae5e96 [id=380e7a7694ec4b019f13d7a649697225]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.776881 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c15e3cec8df74a56adfa7a5d5c1d23a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.776890 3622 leader_election.cc:290] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.781041 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a378e913112c443db869016e3b98fd0e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:42.781090 2377 ts_tablet_manager.cc:1929] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.781203 2377 log.cc:1199] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/17d2cc6b16d34331bb80c576c0cd2ce0
I20251026 02:56:42.781561 2377 ts_tablet_manager.cc:1950] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:42.782580 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 737) took 5030 ms. Trace:
I20251026 02:56:42.782724 2377 rpcz_store.cc:276] 1026 02:56:37.752559 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.752598 (+ 39us) service_pool.cc:225] Handling call
1026 02:56:42.729055 (+4976457us) ts_tablet_manager.cc:1151] Deleting tablet 17d2cc6b16d34331bb80c576c0cd2ce0
1026 02:56:42.729065 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.759346 (+ 30281us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.780988 (+ 21642us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.782562 (+ 1574us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4976331}
I20251026 02:56:42.783001 2381 tablet_bootstrap.cc:492] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:42.783102 2377 tablet_replica.cc:333] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:42.783195 2381 ts_tablet_manager.cc:1403] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.016s
I20251026 02:56:42.783344 2377 raft_consensus.cc:2243] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.783560 2377 raft_consensus.cc:2272] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.784152 2377 ts_tablet_manager.cc:1916] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.785745 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 0afb6f04f5ac463780fe8f522bf2fe23 (DEFAULT_TABLE table=default.Table_94329ed6f5fc42e6b4d1de6b1565adc4 [id=9266e7f9ea024740ae9fca02636d80ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.786162 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0afb6f04f5ac463780fe8f522bf2fe23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.784158 2381 raft_consensus.cc:359] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.791616 2381 raft_consensus.cc:385] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.791700 2381 raft_consensus.cc:740] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.791893 2381 consensus_queue.cc:260] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.792466 2381 ts_tablet_manager.cc:1434] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251026 02:56:42.792735 2381 tablet_bootstrap.cc:492] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:42.795166 2053 consensus_peers.cc:597] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:42.795847 2053 consensus_peers.cc:597] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:42.800036 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44af74033ccf4d048475f12d164938f5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:42.800727 2381 tablet_bootstrap.cc:654] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:42.808821 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 0afb6f04f5ac463780fe8f522bf2fe23 (DEFAULT_TABLE table=default.Table_94329ed6f5fc42e6b4d1de6b1565adc4 [id=9266e7f9ea024740ae9fca02636d80ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.809276 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0afb6f04f5ac463780fe8f522bf2fe23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.810581 2377 ts_tablet_manager.cc:1929] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.810693 2377 log.cc:1199] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/44af74033ccf4d048475f12d164938f5
I20251026 02:56:42.811023 2377 ts_tablet_manager.cc:1950] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:42.811331 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 738) took 5058 ms. Trace:
I20251026 02:56:42.811406 2377 rpcz_store.cc:276] 1026 02:56:37.753247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.753293 (+ 46us) service_pool.cc:225] Handling call
1026 02:56:42.783056 (+5029763us) ts_tablet_manager.cc:1151] Deleting tablet 44af74033ccf4d048475f12d164938f5
1026 02:56:42.783066 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.790487 (+ 7421us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.810445 (+ 19958us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.811311 (+ 866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5029619}
I20251026 02:56:42.811713 2377 tablet_replica.cc:333] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:42.811884 2377 raft_consensus.cc:2243] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.812042 2377 raft_consensus.cc:2272] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.812224 1945 tablet_service.cc:1505] Processing CreateTablet for tablet f840050f7e314d74ac6e604354a7d4c9 (DEFAULT_TABLE table=default.Table_64db0f5adb1740ce80b215181ec835fe [id=087de7421e9f44d481e83b1ba3864d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.812593 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f840050f7e314d74ac6e604354a7d4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.819057 3778 raft_consensus.cc:493] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.819252 3778 raft_consensus.cc:515] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.819934 3778 leader_election.cc:290] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:42.820725 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a3f3fc3c7a04d05bae28472251f3acc" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:42.821319 2053 leader_election.cc:343] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.823269 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6731d8e782234dea862350a78ebbac32" } new_table_name: "default.Table_cbf5e72aa3ea4bcea06df860ce74bb42" modify_external_catalogs: true
I20251026 02:56:42.825109 2376 raft_consensus.cc:2272] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.825724 2376 ts_tablet_manager.cc:1916] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.826983 2377 ts_tablet_manager.cc:1916] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.835184 2376 ts_tablet_manager.cc:1929] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.835407 2376 log.cc:1199] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/0fe0e06a86e241bc91a1db4219fe25bc
I20251026 02:56:42.835958 2376 ts_tablet_manager.cc:1950] T 0fe0e06a86e241bc91a1db4219fe25bc P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:42.836359 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 718) took 4345 ms. Trace:
I20251026 02:56:42.836493 2376 rpcz_store.cc:276] 1026 02:56:38.490628 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.490864 (+ 236us) service_pool.cc:225] Handling call
1026 02:56:42.757250 (+4266386us) ts_tablet_manager.cc:1151] Deleting tablet 0fe0e06a86e241bc91a1db4219fe25bc
1026 02:56:42.757261 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.831875 (+ 74614us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.835080 (+ 3205us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.836342 (+ 1262us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64535,"tablet-delete.queue_time_us":4266192}
I20251026 02:56:42.840703 1654 catalog_manager.cc:5649] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:42.841166 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 882cd1680aa140f6a63a961cd4aec25a (table default.Table_52417f25dbdb4676a5aaaca9bf46962c [id=cc7a8c37a3b342d988872e3baa16a26d]). Expected version 1 got 0
I20251026 02:56:42.841595 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 2fd8f02126b24285bbfa575540ba0f09 (table default.Table_7539d36a088f4eafaa21d151ed9ea1cb [id=1950d7cb826b4a99b1ba8866e1d845f3]). Expected version 1 got 0
I20251026 02:56:42.853530 2377 ts_tablet_manager.cc:1929] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:42.856596 2376 tablet_replica.cc:333] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:42.856846 2376 raft_consensus.cc:2243] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.857023 2376 raft_consensus.cc:2272] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.857574 2376 ts_tablet_manager.cc:1916] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.860456 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 0fe0e06a86e241bc91a1db4219fe25bc (table default.Table_caee34582bea4cd1970c4cc59078f9d1 [id=0494cd534eb24adc94c907471cdb346a]) successfully deleted
I20251026 02:56:42.860723 2377 log.cc:1199] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/a378e913112c443db869016e3b98fd0e
I20251026 02:56:42.862462 2377 ts_tablet_manager.cc:1950] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:42.863276 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 739) took 5109 ms. Trace:
I20251026 02:56:42.866133 2099 tablet_service.cc:1505] Processing CreateTablet for tablet f840050f7e314d74ac6e604354a7d4c9 (DEFAULT_TABLE table=default.Table_64db0f5adb1740ce80b215181ec835fe [id=087de7421e9f44d481e83b1ba3864d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.866600 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f840050f7e314d74ac6e604354a7d4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.866636 2377 rpcz_store.cc:276] 1026 02:56:37.753599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.753636 (+ 37us) service_pool.cc:225] Handling call
1026 02:56:42.811671 (+5058035us) ts_tablet_manager.cc:1151] Deleting tablet a378e913112c443db869016e3b98fd0e
1026 02:56:42.811680 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.839866 (+ 28186us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.853418 (+ 13552us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.863256 (+ 9838us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5057916}
I20251026 02:56:42.866978 2377 tablet_replica.cc:333] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:42.867154 2377 raft_consensus.cc:2243] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 2 LEADER]: Raft consensus shutting down.
I20251026 02:56:42.867389 2377 pending_rounds.cc:70] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:42.867434 2377 pending_rounds.cc:77] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7214888559453450240 op_type: NO_OP noop_request { }
I20251026 02:56:42.867522 2377 raft_consensus.cc:2889] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:42.867594 2377 raft_consensus.cc:2272] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.868790 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 7060cb495eb1423b9e3ecbd17c325948 (DEFAULT_TABLE table=default.Table_9304ae0aeabf48e99b2f10c5fba8cd1e [id=269a726aa2bf481ea34f719266081147]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.869134 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7060cb495eb1423b9e3ecbd17c325948. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.872427 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 87fb7a4d857442fa9a8748ad9004ddb5 (DEFAULT_TABLE table=default.Table_e98c5342f720423ba269cf18b16fcfd9 [id=6ba4dfbf958c4571bfa7fb39cf40eaca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.872786 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87fb7a4d857442fa9a8748ad9004ddb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.873687 2377 ts_tablet_manager.cc:1916] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.882498 2372 ts_tablet_manager.cc:1929] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:42.882767 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 7060cb495eb1423b9e3ecbd17c325948 (DEFAULT_TABLE table=default.Table_9304ae0aeabf48e99b2f10c5fba8cd1e [id=269a726aa2bf481ea34f719266081147]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.883630 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 87fb7a4d857442fa9a8748ad9004ddb5 (DEFAULT_TABLE table=default.Table_e98c5342f720423ba269cf18b16fcfd9 [id=6ba4dfbf958c4571bfa7fb39cf40eaca]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:42.883682 2194 consensus_peers.cc:597] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d90e62ae86b149b291dc29cf7442c6bc. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:42.886104 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7060cb495eb1423b9e3ecbd17c325948. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.886382 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87fb7a4d857442fa9a8748ad9004ddb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.886570 2372 log.cc:1199] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/865fbd674db34f668fb5a2f5836a9b43
I20251026 02:56:42.887030 2372 ts_tablet_manager.cc:1950] T 865fbd674db34f668fb5a2f5836a9b43 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:42.887828 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 700) took 6868 ms. Trace:
I20251026 02:56:42.887982 2372 rpcz_store.cc:276] 1026 02:56:36.018991 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.021826 (+ 2835us) service_pool.cc:225] Handling call
1026 02:56:42.689044 (+6667218us) ts_tablet_manager.cc:1151] Deleting tablet 865fbd674db34f668fb5a2f5836a9b43
1026 02:56:42.689053 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.693525 (+ 4472us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.882351 (+188826us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.887788 (+ 5437us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6638971}
I20251026 02:56:42.888664 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 865fbd674db34f668fb5a2f5836a9b43 (table default.Table_95868643edf84f8a8a325f3cb56aa732 [id=facdc6a612aa47b6bcbef168b5d7f9aa]) successfully deleted
I20251026 02:56:42.889130 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 702) took 6840 ms. Trace:
I20251026 02:56:42.889285 2372 rpcz_store.cc:276] 1026 02:56:36.048962 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.049814 (+ 852us) service_pool.cc:225] Handling call
1026 02:56:42.888991 (+6839177us) ts_tablet_manager.cc:1151] Deleting tablet b8a1244c1c49471fb7711a2ca9b89a99
1026 02:56:42.889001 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.889098 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6763875}
W20251026 02:56:42.889812 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet b8a1244c1c49471fb7711a2ca9b89a99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8a1244c1c49471fb7711a2ca9b89a99 already in progress: creating tablet
I20251026 02:56:42.890219 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 701) took 6860 ms. Trace:
I20251026 02:56:42.890511 2372 rpcz_store.cc:276] 1026 02:56:36.029862 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.066381 (+ 36519us) service_pool.cc:225] Handling call
1026 02:56:42.890089 (+6823708us) ts_tablet_manager.cc:1151] Deleting tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3
1026 02:56:42.890099 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.890187 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6764410}
W20251026 02:56:42.890996 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a9a005ec48c4d24aa1dc141dcf6fdc3 already in progress: creating tablet
I20251026 02:56:42.891360 2372 tablet_replica.cc:333] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:42.891816 2372 raft_consensus.cc:2243] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:42.892038 2372 raft_consensus.cc:2272] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.897128 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 2d6ef860ebd24ee5b2dbd0f0529239be (DEFAULT_TABLE table=default.Table_4236af49f54d455981be3a644c6bb095 [id=0bd661b6266f494d89d20690e9ebb0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.897557 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d6ef860ebd24ee5b2dbd0f0529239be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.899894 2377 ts_tablet_manager.cc:1929] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251026 02:56:42.901923 1975 raft_consensus.cc:3060] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:42.916329 2377 log.cc:1199] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3327d26a2c2c4d0b86d98151f5940bd7
I20251026 02:56:42.909859 3769 raft_consensus.cc:493] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.921332 3769 raft_consensus.cc:515] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.921772 2378 tablet_bootstrap.cc:492] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:42.921916 2378 ts_tablet_manager.cc:1403] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.253s user 0.001s sys 0.002s
I20251026 02:56:42.921998 3769 leader_election.cc:290] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.922852 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:42.923061 1945 tablet_service.cc:1505] Processing CreateTablet for tablet defdf9fb419c41db92f954811dfd4dbb (DEFAULT_TABLE table=default.Table_b83cc3a9c8694aba81d2fcebaaeec62a [id=1f56b6d7178f4484be952224f0e4b7bc]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:42.923460 2194 leader_election.cc:343] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:42.910828 2381 tablet_bootstrap.cc:492] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:42.923695 2381 ts_tablet_manager.cc:1403] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.131s user 0.008s sys 0.013s
I20251026 02:56:42.924468 2381 raft_consensus.cc:359] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.924618 2381 raft_consensus.cc:385] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:42.924669 2381 raft_consensus.cc:740] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:42.925086 2381 consensus_queue.cc:260] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.934481 2381 ts_tablet_manager.cc:1434] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20251026 02:56:42.934868 2381 tablet_bootstrap.cc:492] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:42.925834 3372 raft_consensus.cc:493] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:42.937350 3372 raft_consensus.cc:515] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:42.938035 3372 leader_election.cc:290] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:42.935638 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 65e0f8d848084e48bc672387b0ad0c50 (DEFAULT_TABLE table=default.Table_59ba57e9b91b4b7ca1fefa8880a5cdb7 [id=73ba0053978e4757977a5b1cc1f9b33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.935786 2308 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:42.939621 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e0f8d848084e48bc672387b0ad0c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.940389 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_622b499e485f4f00945466ab4023cc0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:42.901445 2195 leader_election.cc:343] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: d212170f056a44dfb15cfcb7f9fbae2f
I20251026 02:56:42.940845 2195 leader_election.cc:304] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:42.940965 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_622b499e485f4f00945466ab4023cc0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:42.941084 3769 raft_consensus.cc:2749] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:42.941478 2195 consensus_peers.cc:597] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b7d85a2941545ceaa16b22e8328b4ff. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:42.943051 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet defdf9fb419c41db92f954811dfd4dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.897318 3778 raft_consensus.cc:493] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.947736 3778 raft_consensus.cc:515] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.948400 3778 leader_election.cc:290] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:42.949043 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4be3e9d5b07a48999497ed54a3d035fe consensus state has no leader
I20251026 02:56:42.899302 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 2d6ef860ebd24ee5b2dbd0f0529239be (DEFAULT_TABLE table=default.Table_4236af49f54d455981be3a644c6bb095 [id=0bd661b6266f494d89d20690e9ebb0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.949307 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet aa0aa6cdd0de4b0a91f553c2341e9c01 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:42.949515 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d6ef860ebd24ee5b2dbd0f0529239be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.951761 2381 tablet_bootstrap.cc:654] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:42.900521 2083 tablet_service.cc:1505] Processing CreateTablet for tablet defdf9fb419c41db92f954811dfd4dbb (DEFAULT_TABLE table=default.Table_b83cc3a9c8694aba81d2fcebaaeec62a [id=1f56b6d7178f4484be952224f0e4b7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.954129 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet defdf9fb419c41db92f954811dfd4dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.957245 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 17d2cc6b16d34331bb80c576c0cd2ce0 (table default.Table_9f3815b1bf464a83a574e3075d065956 [id=095ad208529348d4aea2d4e2840e07c7]) successfully deleted
I20251026 02:56:42.957568 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 44af74033ccf4d048475f12d164938f5 (table default.Table_e4fbc44faf984982986abfb227d6663d [id=b35a8189bbc04e94bb097c9e13a38ec6]) successfully deleted
I20251026 02:56:42.957751 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:42.957846 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet a378e913112c443db869016e3b98fd0e (table default.Table_f2809b812fe14b2d9e44b22bfa68c1f7 [id=f83014c25cfa449d976ead67854d2ca7]) successfully deleted
I20251026 02:56:42.958496 2372 ts_tablet_manager.cc:1916] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.959096 2376 ts_tablet_manager.cc:1929] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:42.956053 2377 ts_tablet_manager.cc:1950] T 3327d26a2c2c4d0b86d98151f5940bd7 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:42.960063 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d8455ca7030842aa80ba3d8da6c5a632"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:42.960659 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8455ca7030842aa80ba3d8da6c5a632 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:42.962424 2194 consensus_peers.cc:597] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:42.962603 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8ea5a407ed7440bf8bbe18a66503efed" } modify_external_catalogs: true
I20251026 02:56:42.962795 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8ea5a407ed7440bf8bbe18a66503efed" } modify_external_catalogs: true
I20251026 02:56:42.964203 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a636a385c0844582a777f2be1f382310" } new_table_name: "default.Table_b061587f8f69417d870501f300207da4" modify_external_catalogs: true
I20251026 02:56:42.966480 2228 tablet_service.cc:1505] Processing CreateTablet for tablet c15e3cec8df74a56adfa7a5d5c1d23a2 (DEFAULT_TABLE table=default.Table_7175e585b5e549dc89c9f1eebaae5e96 [id=380e7a7694ec4b019f13d7a649697225]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.966652 2242 tablet_service.cc:1505] Processing CreateTablet for tablet f840050f7e314d74ac6e604354a7d4c9 (DEFAULT_TABLE table=default.Table_64db0f5adb1740ce80b215181ec835fe [id=087de7421e9f44d481e83b1ba3864d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.966884 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c15e3cec8df74a56adfa7a5d5c1d23a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.967101 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f840050f7e314d74ac6e604354a7d4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.967165 2233 tablet_service.cc:1505] Processing CreateTablet for tablet defdf9fb419c41db92f954811dfd4dbb (DEFAULT_TABLE table=default.Table_b83cc3a9c8694aba81d2fcebaaeec62a [id=1f56b6d7178f4484be952224f0e4b7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.967439 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet defdf9fb419c41db92f954811dfd4dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.967584 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 0afb6f04f5ac463780fe8f522bf2fe23 (DEFAULT_TABLE table=default.Table_94329ed6f5fc42e6b4d1de6b1565adc4 [id=9266e7f9ea024740ae9fca02636d80ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.967872 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0afb6f04f5ac463780fe8f522bf2fe23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.969902 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 740) took 5213 ms. Trace:
I20251026 02:56:42.969977 2377 rpcz_store.cc:276] 1026 02:56:37.756145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.756193 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:42.866929 (+5110736us) ts_tablet_manager.cc:1151] Deleting tablet 3327d26a2c2c4d0b86d98151f5940bd7
1026 02:56:42.866939 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.896635 (+ 29696us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.899781 (+ 3146us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.969883 (+ 70102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5110551}
I20251026 02:56:42.970312 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 741) took 5213 ms. Trace:
I20251026 02:56:42.970387 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 65e0f8d848084e48bc672387b0ad0c50 (DEFAULT_TABLE table=default.Table_59ba57e9b91b4b7ca1fefa8880a5cdb7 [id=73ba0053978e4757977a5b1cc1f9b33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.970369 2377 rpcz_store.cc:276] 1026 02:56:37.756487 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.756528 (+ 41us) service_pool.cc:225] Handling call
1026 02:56:42.970214 (+5213686us) ts_tablet_manager.cc:1151] Deleting tablet 8aa0575af07e433c8d96acab31edb4af
1026 02:56:42.970222 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.970300 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5213547}
I20251026 02:56:42.970773 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e0f8d848084e48bc672387b0ad0c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.971006 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 725) took 5220 ms. Trace:
I20251026 02:56:42.971094 2377 rpcz_store.cc:276] 1026 02:56:37.750531 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.756653 (+ 6122us) service_pool.cc:225] Handling call
1026 02:56:42.970853 (+5214200us) ts_tablet_manager.cc:1151] Deleting tablet b8cad7adae7e43469ceadbedeee53564
1026 02:56:42.970864 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.970943 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5214092}
I20251026 02:56:42.971516 2377 tablet_replica.cc:333] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:42.971700 2377 raft_consensus.cc:2243] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:42.972035 2377 raft_consensus.cc:2272] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:42.972337 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a3f3fc3c7a04d05bae28472251f3acc" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:42.972612 2377 ts_tablet_manager.cc:1916] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:42.972859 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 7060cb495eb1423b9e3ecbd17c325948 (DEFAULT_TABLE table=default.Table_9304ae0aeabf48e99b2f10c5fba8cd1e [id=269a726aa2bf481ea34f719266081147]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.973199 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7060cb495eb1423b9e3ecbd17c325948. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.976514 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 4835ee8bfb764df9a19f9220b7f8e152 (DEFAULT_TABLE table=default.Table_4e452bf0486241589709e97602ea9942 [id=d621b84f806e4575be7b7c9895e66779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.976639 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 2d6ef860ebd24ee5b2dbd0f0529239be (DEFAULT_TABLE table=default.Table_4236af49f54d455981be3a644c6bb095 [id=0bd661b6266f494d89d20690e9ebb0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.976868 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4835ee8bfb764df9a19f9220b7f8e152. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.977093 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d6ef860ebd24ee5b2dbd0f0529239be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.977656 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_da48008b6d1a4e0b8dd4f80c2bfad18a" } modify_external_catalogs: true
I20251026 02:56:42.977859 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_da48008b6d1a4e0b8dd4f80c2bfad18a" } modify_external_catalogs: true
W20251026 02:56:42.978056 2195 leader_election.cc:343] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: a378e913112c443db869016e3b98fd0e
I20251026 02:56:42.978207 2195 leader_election.cc:304] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:42.978375 2195 raft_consensus.cc:2686] T a378e913112c443db869016e3b98fd0e P 696476265495492790739ec126a08539: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251026 02:56:42.978968 2195 leader_election.cc:343] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 44af74033ccf4d048475f12d164938f5
I20251026 02:56:42.980302 2195 leader_election.cc:304] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:42.980598 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 87fb7a4d857442fa9a8748ad9004ddb5 (DEFAULT_TABLE table=default.Table_e98c5342f720423ba269cf18b16fcfd9 [id=6ba4dfbf958c4571bfa7fb39cf40eaca]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:42.981276 2195 raft_consensus.cc:2686] T 44af74033ccf4d048475f12d164938f5 P 696476265495492790739ec126a08539: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251026 02:56:42.981432 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87fb7a4d857442fa9a8748ad9004ddb5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:42.987608 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12e10fd463cd43c7abb22ef428c0aa37 consensus state has no leader
I20251026 02:56:42.991781 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c062fd73dfee44db8acf3124482a05ed" } modify_external_catalogs: true
I20251026 02:56:42.991964 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c062fd73dfee44db8acf3124482a05ed" } modify_external_catalogs: true
I20251026 02:56:42.961819 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 4835ee8bfb764df9a19f9220b7f8e152 (DEFAULT_TABLE table=default.Table_4e452bf0486241589709e97602ea9942 [id=d621b84f806e4575be7b7c9895e66779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.992352 2376 log.cc:1199] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/0de06ea2b0544402a979d9ddc487860f
I20251026 02:56:42.992628 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4835ee8bfb764df9a19f9220b7f8e152. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.992664 2376 ts_tablet_manager.cc:1950] T 0de06ea2b0544402a979d9ddc487860f P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:42.992983 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 720) took 4502 ms. Trace:
I20251026 02:56:42.993055 2376 rpcz_store.cc:276] 1026 02:56:38.490815 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.490859 (+ 44us) service_pool.cc:225] Handling call
1026 02:56:42.837009 (+4346150us) ts_tablet_manager.cc:1151] Deleting tablet 0de06ea2b0544402a979d9ddc487860f
1026 02:56:42.844862 (+ 7853us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.892961 (+ 48099us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.958995 (+ 66034us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:42.992965 (+ 33970us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":402,"tablet-delete.queue_time_us":4345950}
I20251026 02:56:42.993741 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5a7f5adeea764cda87c60443f2e29e50" } modify_external_catalogs: true
I20251026 02:56:42.993892 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5a7f5adeea764cda87c60443f2e29e50" } modify_external_catalogs: true
I20251026 02:56:42.996728 3769 raft_consensus.cc:493] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:42.996866 3769 raft_consensus.cc:515] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:42.997473 3769 leader_election.cc:290] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:43.005327 3778 raft_consensus.cc:493] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.005493 3778 raft_consensus.cc:515] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.006093 3778 leader_election.cc:290] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:42.936580 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 65e0f8d848084e48bc672387b0ad0c50 (DEFAULT_TABLE table=default.Table_59ba57e9b91b4b7ca1fefa8880a5cdb7 [id=73ba0053978e4757977a5b1cc1f9b33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.961054 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 4835ee8bfb764df9a19f9220b7f8e152 (DEFAULT_TABLE table=default.Table_4e452bf0486241589709e97602ea9942 [id=d621b84f806e4575be7b7c9895e66779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.936934 3841 raft_consensus.cc:493] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.010000 3841 raft_consensus.cc:515] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.010164 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e0f8d848084e48bc672387b0ad0c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.010362 2118 raft_consensus.cc:2468] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:43.010632 3841 leader_election.cc:290] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.011265 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4835ee8bfb764df9a19f9220b7f8e152. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:42.988797 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95c9fbae62db4850b0f02a2443a3432b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:43.012274 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 658c52c62c854c07b98fb443ee60bd0e (DEFAULT_TABLE table=default.Table_2c3a87b75efb47f28113790cea7ac82d [id=c3046ca9b0264d4d80c1a6381136f0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:42.998078 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec13c5386abe414ba25589e03d5b850a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:43.012640 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 658c52c62c854c07b98fb443ee60bd0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.012653 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 0de06ea2b0544402a979d9ddc487860f (table default.Table_6546379855b749b889279973db1820cd [id=b8be7adba1cd48ae979de506d892cc3d]) successfully deleted
W20251026 02:56:43.012873 2194 leader_election.cc:343] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.005959 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 658c52c62c854c07b98fb443ee60bd0e (DEFAULT_TABLE table=default.Table_2c3a87b75efb47f28113790cea7ac82d [id=c3046ca9b0264d4d80c1a6381136f0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.021143 1956 tablet_service.cc:1505] Processing CreateTablet for tablet f73d8fe8d23d47c980d7de00aaa47031 (DEFAULT_TABLE table=default.Table_9313700399784167b4022a0e6e3a56f7 [id=6059abc79c0f409aa40699f85b1b1836]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.024768 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 097cc2f7f7b946988194abdde2d803d1 (DEFAULT_TABLE table=default.Table_9e3fab0107274734a5532679436e4931 [id=35d5da0e34e44e9dacbf995fd2ca3f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.024828 2096 tablet_service.cc:1505] Processing CreateTablet for tablet f73d8fe8d23d47c980d7de00aaa47031 (DEFAULT_TABLE table=default.Table_9313700399784167b4022a0e6e3a56f7 [id=6059abc79c0f409aa40699f85b1b1836]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.025293 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73d8fe8d23d47c980d7de00aaa47031. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:43.029543 1916 leader_election.cc:343] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: a378e913112c443db869016e3b98fd0e
I20251026 02:56:43.032147 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 2b4bc725e76c41bcab92e87c9b85f95d (DEFAULT_TABLE table=default.Table_279edde3f26a49bcbf0c6c501709fadc [id=ecabe158d73b4a94916ff1e4bff1b63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.033785 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 2b4bc725e76c41bcab92e87c9b85f95d (DEFAULT_TABLE table=default.Table_279edde3f26a49bcbf0c6c501709fadc [id=ecabe158d73b4a94916ff1e4bff1b63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.034173 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b4bc725e76c41bcab92e87c9b85f95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.034873 1948 tablet_service.cc:1505] Processing CreateTablet for tablet c56c26546da84d599544208ec428dbe4 (DEFAULT_TABLE table=default.Table_675403100cd74e78a423ffac082f4e8b [id=252009a6d04a4c21a2925c6fe4d6859b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.036475 1657 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_dabd227b99964351870151b7731b8175 [id=604570227c014f66aeae89d93edaaafe] alter complete (version 1)
I20251026 02:56:43.036517 2097 tablet_service.cc:1505] Processing CreateTablet for tablet c56c26546da84d599544208ec428dbe4 (DEFAULT_TABLE table=default.Table_675403100cd74e78a423ffac082f4e8b [id=252009a6d04a4c21a2925c6fe4d6859b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.036934 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c56c26546da84d599544208ec428dbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.038390 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4c6dfbbb86f14ddb8ca1ae3b8a51f70b" } modify_external_catalogs: true
I20251026 02:56:43.038594 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4c6dfbbb86f14ddb8ca1ae3b8a51f70b" } modify_external_catalogs: true
I20251026 02:56:43.039237 2372 ts_tablet_manager.cc:1929] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:43.039330 2372 log.cc:1199] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/30c619ad37364747b6ebf97a73ef45a2
I20251026 02:56:43.040776 2095 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b7d85a2941545ceaa16b22e8328b4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.040872 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 99d318aa0d1847c58167bf0af9e75aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.040906 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a345fa5edbd48d2891b8b40b09cb3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.041015 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet a6f9ae327aa4401baf66a737eaa93cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.044720 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9b0e2338f40441e6b644b1e7740df97b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.045710 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b0e2338f40441e6b644b1e7740df97b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.047717 3372 raft_consensus.cc:493] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:43.047863 3372 raft_consensus.cc:515] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:43.047957 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 69034.2us
I20251026 02:56:43.048503 3372 leader_election.cc:290] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:43.049168 1916 leader_election.cc:343] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 44af74033ccf4d048475f12d164938f5
I20251026 02:56:43.050348 2381 tablet_bootstrap.cc:492] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:43.050558 2381 ts_tablet_manager.cc:1403] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.116s user 0.004s sys 0.000s
I20251026 02:56:43.051802 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 097cc2f7f7b946988194abdde2d803d1 (DEFAULT_TABLE table=default.Table_9e3fab0107274734a5532679436e4931 [id=35d5da0e34e44e9dacbf995fd2ca3f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.052150 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 097cc2f7f7b946988194abdde2d803d1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:43.053061 2194 consensus_peers.cc:597] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:43.051319 2381 raft_consensus.cc:359] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.054255 2381 raft_consensus.cc:385] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.054365 2381 raft_consensus.cc:740] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.054620 2381 consensus_queue.cc:260] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.060696 2377 ts_tablet_manager.cc:1929] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:43.061686 2377 log.cc:1199] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/6108219254bd448c852096c0c967384b
I20251026 02:56:43.063510 2378 raft_consensus.cc:359] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:43.063678 2378 raft_consensus.cc:385] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.063727 2378 raft_consensus.cc:740] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
W20251026 02:56:43.064762 1650 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.61:37655, user_credentials={real_user=slave}} blocked reactor thread for 58716.9us
I20251026 02:56:43.065316 2377 ts_tablet_manager.cc:1950] T 6108219254bd448c852096c0c967384b P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:43.071873 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 721) took 4441 ms. Trace:
I20251026 02:56:43.072059 2376 rpcz_store.cc:276] 1026 02:56:38.629943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.630286 (+ 343us) service_pool.cc:225] Handling call
1026 02:56:43.012886 (+4382600us) ts_tablet_manager.cc:1151] Deleting tablet 0cf76ef8f6af445eb0ce15c8d79136c5
1026 02:56:43.069243 (+ 56357us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.071847 (+ 2604us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4382387}
I20251026 02:56:43.071815 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 727) took 5320 ms. Trace:
W20251026 02:56:43.072845 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0cf76ef8f6af445eb0ce15c8d79136c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cf76ef8f6af445eb0ce15c8d79136c5 already in progress: creating tablet
I20251026 02:56:43.073674 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 658c52c62c854c07b98fb443ee60bd0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.072670 2377 rpcz_store.cc:276] 1026 02:56:37.750682 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.756786 (+ 6104us) service_pool.cc:225] Handling call
1026 02:56:42.971439 (+5214653us) ts_tablet_manager.cc:1151] Deleting tablet 6108219254bd448c852096c0c967384b
1026 02:56:42.971447 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:42.975565 (+ 4118us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.059877 (+ 84312us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.071559 (+ 11682us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5214505}
I20251026 02:56:43.075156 2375 tablet_bootstrap.cc:654] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.075511 2377 tablet_replica.cc:333] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:43.075973 2378 consensus_queue.cc:260] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:43.076416 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73d8fe8d23d47c980d7de00aaa47031. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.076748 2378 ts_tablet_manager.cc:1434] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.155s user 0.001s sys 0.001s
I20251026 02:56:43.076972 2377 raft_consensus.cc:2243] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 2 LEADER]: Raft consensus shutting down.
I20251026 02:56:43.079125 2377 raft_consensus.cc:2272] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.084059 1655 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_7244d7be0e764416aaba258478bdffdc [id=87349d9c6c764382b828e716e7854c59] alter complete (version 4)
I20251026 02:56:43.083770 2377 ts_tablet_manager.cc:1916] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:43.088536 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 80150us
W20251026 02:56:43.089722 2053 leader_election.cc:343] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.090134 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet ff242ef9541a42d793c4484190956da6 (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]). Expected version 1 got 0
I20251026 02:56:43.090543 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b7298917793f498a8273bb74124d7807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.092666 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7298917793f498a8273bb74124d7807 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.094206 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet a6f9ae327aa4401baf66a737eaa93cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:43.096468 2381 ts_tablet_manager.cc:1434] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20251026 02:56:43.096642 3841 raft_consensus.cc:493] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.096792 3841 raft_consensus.cc:515] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.097386 3841 leader_election.cc:290] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.107247 1960 tablet_service.cc:1505] Processing CreateTablet for tablet ac82c14e7dcb448b845d998249e2b339 (DEFAULT_TABLE table=default.Table_464e60af830f4416af8fdcf84f660a49 [id=de836edc225f41d8b0c68748936306ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.108801 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cbd8bd7117a34f5a836578a05d997586" } new_table_name: "default.Table_301efb6c96d4459994003a291f60cded" modify_external_catalogs: true
I20251026 02:56:43.111264 2381 tablet_bootstrap.cc:492] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:43.112898 2381 tablet_bootstrap.cc:654] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.114444 2083 tablet_service.cc:1505] Processing CreateTablet for tablet ac82c14e7dcb448b845d998249e2b339 (DEFAULT_TABLE table=default.Table_464e60af830f4416af8fdcf84f660a49 [id=de836edc225f41d8b0c68748936306ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.114578 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3327d26a2c2c4d0b86d98151f5940bd7 (table default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 [id=7de20c0869584b56b936144d62202444]) successfully deleted
W20251026 02:56:43.114991 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 8aa0575af07e433c8d96acab31edb4af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aa0575af07e433c8d96acab31edb4af already in progress: creating tablet
W20251026 02:56:43.114993 1915 leader_election.cc:343] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: a378e913112c443db869016e3b98fd0e
I20251026 02:56:43.115099 1915 leader_election.cc:304] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
W20251026 02:56:43.115231 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet b8cad7adae7e43469ceadbedeee53564 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8cad7adae7e43469ceadbedeee53564 already in progress: creating tablet
W20251026 02:56:43.115329 2053 leader_election.cc:343] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.115440 2053 leader_election.cc:304] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.115517 3372 raft_consensus.cc:2749] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:43.115734 3849 raft_consensus.cc:2749] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.115962 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac82c14e7dcb448b845d998249e2b339. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.116366 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:43.116680 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f22c6c163c34ed48a0b7eeb5a574e76" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:43.116880 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e38f282e8414b7db2a6d5f4a8933e84" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:43.117229 2053 leader_election.cc:343] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.117533 2378 tablet_bootstrap.cc:492] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:43.117965 2053 leader_election.cc:343] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:43.118227 2053 leader_election.cc:343] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.118908 2378 tablet_bootstrap.cc:654] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.119371 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 6108219254bd448c852096c0c967384b (table default.Table_5af1941b0082444c8bd348a080af20ee [id=6342ae7b8acf4137a16ebf978ce7e15b]) successfully deleted
I20251026 02:56:43.113104 3769 consensus_queue.cc:1048] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:43.113406 3537 consensus_queue.cc:1048] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:43.128136 2372 ts_tablet_manager.cc:1950] T 30c619ad37364747b6ebf97a73ef45a2 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:43.134557 3843 raft_consensus.cc:493] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.134588 3842 raft_consensus.cc:493] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.135047 3842 raft_consensus.cc:515] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.135627 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec13c5386abe414ba25589e03d5b850a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:43.135785 2118 raft_consensus.cc:2468] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:43.136428 3842 leader_election.cc:290] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.135025 3843 raft_consensus.cc:515] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.137687 3843 leader_election.cc:290] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.126477 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac82c14e7dcb448b845d998249e2b339. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.126706 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 723) took 4486 ms. Trace:
I20251026 02:56:43.126839 3816 tablet.cc:1722] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:43.146267 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 097cc2f7f7b946988194abdde2d803d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.146683 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b4bc725e76c41bcab92e87c9b85f95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.152568 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c56c26546da84d599544208ec428dbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.157464 2375 tablet_bootstrap.cc:492] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:43.157918 2376 rpcz_store.cc:276] 1026 02:56:38.639796 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.639851 (+ 55us) service_pool.cc:225] Handling call
1026 02:56:43.072671 (+4432820us) ts_tablet_manager.cc:1151] Deleting tablet 2f1872ef9f244a388d25a8085dd53f82
1026 02:56:43.126540 (+ 53869us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.126691 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4432661}
I20251026 02:56:43.158944 2377 ts_tablet_manager.cc:1929] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251026 02:56:43.160036 2377 log.cc:1199] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/cfc3aed64ad0439cb22fd6815c973d87
I20251026 02:56:43.140866 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 6403483a7e0f4676a8a88ce55beeae80 (DEFAULT_TABLE table=default.Table_de99eb535be548db8335e7fbcfe8adb7 [id=ea3b1f4c26e842b28c9bb9b9341439d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.160807 2375 ts_tablet_manager.cc:1403] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.432s user 0.001s sys 0.003s
I20251026 02:56:43.161708 2375 raft_consensus.cc:359] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.161950 2375 raft_consensus.cc:385] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.162068 2375 raft_consensus.cc:740] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.162388 2375 consensus_queue.cc:260] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.163010 2375 ts_tablet_manager.cc:1434] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:43.163311 2375 tablet_bootstrap.cc:492] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:43.163307 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 725) took 4513 ms. Trace:
I20251026 02:56:43.163419 2376 rpcz_store.cc:276] 1026 02:56:38.650032 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.650158 (+ 126us) service_pool.cc:225] Handling call
1026 02:56:43.158490 (+4508332us) ts_tablet_manager.cc:1151] Deleting tablet 62d30d79a6e4432a8dbe40a104e87fa0
1026 02:56:43.163150 (+ 4660us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.163284 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4508083}
I20251026 02:56:43.163789 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 726) took 4513 ms. Trace:
I20251026 02:56:43.163866 2376 rpcz_store.cc:276] 1026 02:56:38.650110 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.650153 (+ 43us) service_pool.cc:225] Handling call
1026 02:56:43.163684 (+4513531us) ts_tablet_manager.cc:1151] Deleting tablet 5c6e3e47b3e7421e945cfbe671b06abe
1026 02:56:43.163694 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.163774 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21,"tablet-delete.queue_time_us":4513278}
I20251026 02:56:43.164160 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 724) took 4514 ms. Trace:
I20251026 02:56:43.164273 2376 rpcz_store.cc:276] 1026 02:56:38.649936 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.655237 (+ 5301us) service_pool.cc:225] Handling call
1026 02:56:43.164077 (+4508840us) ts_tablet_manager.cc:1151] Deleting tablet 3ac38e6257e74367b06b15fc49d2af81
1026 02:56:43.164085 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.164147 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4508572}
I20251026 02:56:43.164556 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 727) took 4507 ms. Trace:
I20251026 02:56:43.164382 2377 ts_tablet_manager.cc:1950] T cfc3aed64ad0439cb22fd6815c973d87 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:43.164630 2376 rpcz_store.cc:276] 1026 02:56:38.656962 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.675865 (+ 18903us) service_pool.cc:225] Handling call
1026 02:56:43.164469 (+4488604us) ts_tablet_manager.cc:1151] Deleting tablet 863aef239cd34a358a71ddfdba6fb72a
1026 02:56:43.164477 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.164543 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4488409}
I20251026 02:56:43.164844 2376 tablet_replica.cc:333] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:43.165030 2376 raft_consensus.cc:2243] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:43.165189 2376 raft_consensus.cc:2272] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.165745 2376 ts_tablet_manager.cc:1916] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:43.164769 3853 tablet.cc:1722] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:43.168468 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41b7b0c36ba748f0a1565892dbe98a78" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:43.169215 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f437d8b5f5784e8fbec10d0700e90348" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:43.169452 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6403483a7e0f4676a8a88ce55beeae80. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:43.170456 2053 consensus_peers.cc:597] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:43.170637 2053 leader_election.cc:343] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:43.170773 2053 leader_election.cc:343] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.171151 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 658c52c62c854c07b98fb443ee60bd0e (DEFAULT_TABLE table=default.Table_2c3a87b75efb47f28113790cea7ac82d [id=c3046ca9b0264d4d80c1a6381136f0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.171448 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:25 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.171643 2228 tablet_service.cc:1505] Processing CreateTablet for tablet f73d8fe8d23d47c980d7de00aaa47031 (DEFAULT_TABLE table=default.Table_9313700399784167b4022a0e6e3a56f7 [id=6059abc79c0f409aa40699f85b1b1836]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.171715 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet d212170f056a44dfb15cfcb7f9fbae2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.171645 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet c31d2704d1f14ddead69c794cc631ec1 with delete_type TABLET_DATA_DELETED (replaced by tablet de9124fe1c8945d9b9565cb732bef20e) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.171645 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet bfa68f2c942c43e2a02e9d333bdfa2a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 262f60fdb274476abf3533b4f47688b8) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.172034 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 097cc2f7f7b946988194abdde2d803d1 (DEFAULT_TABLE table=default.Table_9e3fab0107274734a5532679436e4931 [id=35d5da0e34e44e9dacbf995fd2ca3f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.172142 2240 tablet_service.cc:1505] Processing CreateTablet for tablet ac82c14e7dcb448b845d998249e2b339 (DEFAULT_TABLE table=default.Table_464e60af830f4416af8fdcf84f660a49 [id=de836edc225f41d8b0c68748936306ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.172463 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95c9fbae62db4850b0f02a2443a3432b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.172516 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 2b4bc725e76c41bcab92e87c9b85f95d (DEFAULT_TABLE table=default.Table_279edde3f26a49bcbf0c6c501709fadc [id=ecabe158d73b4a94916ff1e4bff1b63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.172935 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 21b936fa4f6a4ff6af321971a30cccbe with delete_type TABLET_DATA_DELETED (replaced by tablet a956ebfb1eaa4d6fb1575b4165293193) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.173005 2229 tablet_service.cc:1505] Processing CreateTablet for tablet c56c26546da84d599544208ec428dbe4 (DEFAULT_TABLE table=default.Table_675403100cd74e78a423ffac082f4e8b [id=252009a6d04a4c21a2925c6fe4d6859b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.173300 2227 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e38f282e8414b7db2a6d5f4a8933e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e0b929f603b4c25b5b971ee9db683e8) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.173466 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet ec13c5386abe414ba25589e03d5b850a with delete_type TABLET_DATA_DELETED (replaced by tablet 6699b50d87964f6a89c365171253ad25) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.176368 2376 ts_tablet_manager.cc:1929] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:43.178467 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e38f282e8414b7db2a6d5f4a8933e84" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.178647 2259 raft_consensus.cc:2468] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:43.178987 3622 raft_consensus.cc:493] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.179152 3622 raft_consensus.cc:515] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.179745 3622 leader_election.cc:290] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:43.181708 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 19918cb3b7d0450f9de17026f8ea3295
W20251026 02:56:43.183095 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 2f1872ef9f244a388d25a8085dd53f82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f1872ef9f244a388d25a8085dd53f82 already in progress: creating tablet
I20251026 02:56:43.184432 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 658c52c62c854c07b98fb443ee60bd0e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:43.184399 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 62d30d79a6e4432a8dbe40a104e87fa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62d30d79a6e4432a8dbe40a104e87fa0 already in progress: creating tablet
I20251026 02:56:43.184960 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac82c14e7dcb448b845d998249e2b339. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:43.184954 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 5c6e3e47b3e7421e945cfbe671b06abe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c6e3e47b3e7421e945cfbe671b06abe already in progress: creating tablet
I20251026 02:56:43.185323 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 097cc2f7f7b946988194abdde2d803d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.185691 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73d8fe8d23d47c980d7de00aaa47031. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.186031 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c56c26546da84d599544208ec428dbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.186399 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b4bc725e76c41bcab92e87c9b85f95d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:43.186417 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 863aef239cd34a358a71ddfdba6fb72a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 863aef239cd34a358a71ddfdba6fb72a already in progress: creating tablet
I20251026 02:56:43.188143 3537 raft_consensus.cc:493] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.188311 3537 raft_consensus.cc:515] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.188856 3537 leader_election.cc:290] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:43.189175 2375 tablet_bootstrap.cc:654] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:43.194679 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa1253c60ffb4b37849c9318307b94f1 consensus state has no leader
I20251026 02:56:43.195132 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_251030404f2042a984d8c0dacdf3d560"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:43.195297 1661 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet ff242ef9541a42d793c4484190956da6 (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]). Expected version 1 got 0
W20251026 02:56:43.195802 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_251030404f2042a984d8c0dacdf3d560 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.196681 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fde9bb8a8be244ed9462ed7367173e96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.199662 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fde9bb8a8be244ed9462ed7367173e96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.200882 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 19918cb3b7d0450f9de17026f8ea3295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.160807 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 699) took 7127 ms. Trace:
I20251026 02:56:43.201871 2372 rpcz_store.cc:276] 1026 02:56:36.018852 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.022472 (+ 3620us) service_pool.cc:225] Handling call
1026 02:56:42.891300 (+6868828us) ts_tablet_manager.cc:1151] Deleting tablet 30c619ad37364747b6ebf97a73ef45a2
1026 02:56:42.891310 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.027982 (+136672us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.039123 (+ 11141us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.145865 (+106742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6761016}
I20251026 02:56:43.202464 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 30c619ad37364747b6ebf97a73ef45a2 (table default.Table_ad4b4db01c7d4fe0a4e99f2f54cee0e8 [id=65a13a0482834ea6b52b71c953d61e5d]) successfully deleted
I20251026 02:56:43.202756 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.202919 2259 raft_consensus.cc:2468] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:43.204381 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 12e10fd463cd43c7abb22ef428c0aa37 (DEFAULT_TABLE table=default.Table_cbf5e72aa3ea4bcea06df860ce74bb42 [id=b31c19a4727a4647a8bab645f5c784f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.204444 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f22c6c163c34ed48a0b7eeb5a574e76" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.204870 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e10fd463cd43c7abb22ef428c0aa37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.205456 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41b7b0c36ba748f0a1565892dbe98a78" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.205646 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 4984a3aea6a245ffba7f393b47add7d0 (DEFAULT_TABLE table=default.Table_efc39d27d9d5438ea04cef669a110c8a [id=2494ffc89efd452d9778517b731a3dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.205963 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4984a3aea6a245ffba7f393b47add7d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.206035 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f437d8b5f5784e8fbec10d0700e90348" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.206926 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 6403483a7e0f4676a8a88ce55beeae80 (DEFAULT_TABLE table=default.Table_de99eb535be548db8335e7fbcfe8adb7 [id=ea3b1f4c26e842b28c9bb9b9341439d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.207208 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6403483a7e0f4676a8a88ce55beeae80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.209028 2085 tablet_service.cc:1505] Processing CreateTablet for tablet e4c149e8b7f9420c8dba56da2e21f3f9 (DEFAULT_TABLE table=default.Table_0d7b9434a0494a618c25ceebd7f65de8 [id=bb7b13116ad94fa9bd15a1cdb15d207b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.209362 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4c149e8b7f9420c8dba56da2e21f3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.212589 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 716007b9d96e43699033e8a5879afcac (DEFAULT_TABLE table=default.Table_52557bc43e124635b11e4e542e6f62df [id=8f7c804d94984a0b9451c3e1e86d79da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.213004 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 716007b9d96e43699033e8a5879afcac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.219116 3842 raft_consensus.cc:493] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.219266 3842 raft_consensus.cc:515] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.219820 3842 leader_election.cc:290] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.223038 2082 tablet_service.cc:1505] Processing CreateTablet for tablet e45600a92f8e46b096badd700779856d (DEFAULT_TABLE table=default.Table_81d15a2f4d424910a51421472c159f0d [id=0f4a4be6bea7411aa2e13984864e827c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.223573 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e45600a92f8e46b096badd700779856d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.224436 1663 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet ff242ef9541a42d793c4484190956da6 (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]). Expected version 1 got 0
I20251026 02:56:43.226977 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 95c9fbae62db4850b0f02a2443a3432b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.229096 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21b936fa4f6a4ff6af321971a30cccbe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:43.229315 2111 raft_consensus.cc:2468] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:43.229997 2111 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe48472efa444bf580e4074c96a9de92" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:43.230147 2111 raft_consensus.cc:2468] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:43.240157 2053 leader_election.cc:343] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.240417 2053 leader_election.cc:304] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.240691 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0245af5a2dcb4f569b62307965895c1f
I20251026 02:56:43.240676 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 729) took 5489 ms. Trace:
I20251026 02:56:43.240783 2377 rpcz_store.cc:276] 1026 02:56:37.750949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.757013 (+ 6064us) service_pool.cc:225] Handling call
1026 02:56:43.075035 (+5318022us) ts_tablet_manager.cc:1151] Deleting tablet cfc3aed64ad0439cb22fd6815c973d87
1026 02:56:43.075262 (+ 227us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.116500 (+ 41238us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.157579 (+ 41079us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.240651 (+ 83072us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5317837}
I20251026 02:56:43.241289 3842 raft_consensus.cc:2749] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.240535 2376 log.cc:1199] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/efd751ee4b1d433198dcd00c081f648d
I20251026 02:56:43.241588 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 734) took 5489 ms. Trace:
I20251026 02:56:43.241791 2376 ts_tablet_manager.cc:1950] T efd751ee4b1d433198dcd00c081f648d P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:43.241796 2377 rpcz_store.cc:276] 1026 02:56:37.751945 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.757218 (+ 5273us) service_pool.cc:225] Handling call
1026 02:56:43.241335 (+5484117us) ts_tablet_manager.cc:1151] Deleting tablet 7f5cd37c87f34a40bf05829af6de0473
1026 02:56:43.241379 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.241568 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5483994}
I20251026 02:56:43.242161 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 868dbbfce40c4dd6b36cc220966b1986
I20251026 02:56:43.242499 2377 tablet_replica.cc:333] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:43.242700 2053 leader_election.cc:304] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.242962 3842 raft_consensus.cc:2804] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:43.242959 2377 raft_consensus.cc:2243] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:43.243031 3842 raft_consensus.cc:493] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:43.243086 3842 raft_consensus.cc:3060] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.244525 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9b9dc0825e0844f68d5afcc201db737c
I20251026 02:56:43.245647 3842 raft_consensus.cc:515] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.246243 3842 leader_election.cc:290] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.247224 2053 leader_election.cc:304] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.247512 3842 raft_consensus.cc:2804] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:43.247771 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet bcd5621aa3754733b22508be89f080db from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:43.247617 3842 raft_consensus.cc:493] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:43.247936 3842 raft_consensus.cc:3060] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.250772 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_94329ed6f5fc42e6b4d1de6b1565adc4" } new_table_name: "default.Table_6288384ccff04bfca7c4b0a988abfc09" modify_external_catalogs: true
I20251026 02:56:43.250927 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 728) took 4557 ms. Trace:
I20251026 02:56:43.250981 3842 raft_consensus.cc:515] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.252252 3849 raft_consensus.cc:493] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.252372 3849 raft_consensus.cc:515] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.253255 3849 leader_election.cc:290] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.253616 2195 leader_election.cc:304] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.189690 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe48472efa444bf580e4074c96a9de92" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:43.206626 1915 leader_election.cc:343] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 44af74033ccf4d048475f12d164938f5
I20251026 02:56:43.254451 1915 leader_election.cc:304] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
W20251026 02:56:43.254741 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 48131.7us
I20251026 02:56:43.254910 3372 raft_consensus.cc:2749] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:43.255240 3842 leader_election.cc:290] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.255402 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b9dc0825e0844f68d5afcc201db737c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:43.222057 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 7801f073d5d94076af9299492f86fad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:43.222209 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 19918cb3b7d0450f9de17026f8ea3295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:43.236116 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 1104345f80d6418fa907a8a26924c281 (DEFAULT_TABLE table=default.Table_c985fc44cad2462f8e82eadf887551c8 [id=09dc650467064e92a056f6344fda1c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.255882 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1104345f80d6418fa907a8a26924c281. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:43.257792 2053 leader_election.cc:343] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.236898 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 12e10fd463cd43c7abb22ef428c0aa37 (DEFAULT_TABLE table=default.Table_cbf5e72aa3ea4bcea06df860ce74bb42 [id=b31c19a4727a4647a8bab645f5c784f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.259860 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e10fd463cd43c7abb22ef428c0aa37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.183315 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21b936fa4f6a4ff6af321971a30cccbe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:43.238261 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 716007b9d96e43699033e8a5879afcac (DEFAULT_TABLE table=default.Table_52557bc43e124635b11e4e542e6f62df [id=8f7c804d94984a0b9451c3e1e86d79da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.238562 1945 tablet_service.cc:1505] Processing CreateTablet for tablet e4c149e8b7f9420c8dba56da2e21f3f9 (DEFAULT_TABLE table=default.Table_0d7b9434a0494a618c25ceebd7f65de8 [id=bb7b13116ad94fa9bd15a1cdb15d207b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.239675 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe48472efa444bf580e4074c96a9de92" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:43.261094 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 716007b9d96e43699033e8a5879afcac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.261487 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 4984a3aea6a245ffba7f393b47add7d0 (DEFAULT_TABLE table=default.Table_efc39d27d9d5438ea04cef669a110c8a [id=2494ffc89efd452d9778517b731a3dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.261792 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4984a3aea6a245ffba7f393b47add7d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.237579 1948 tablet_service.cc:1505] Processing CreateTablet for tablet e45600a92f8e46b096badd700779856d (DEFAULT_TABLE table=default.Table_81d15a2f4d424910a51421472c159f0d [id=0f4a4be6bea7411aa2e13984864e827c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.252548 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 0245af5a2dcb4f569b62307965895c1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:43.264492 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4c149e8b7f9420c8dba56da2e21f3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.253031 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet 868dbbfce40c4dd6b36cc220966b1986 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:43.252390 1976 raft_consensus.cc:3060] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.270105 2378 tablet_bootstrap.cc:492] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:43.270232 2378 ts_tablet_manager.cc:1403] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.153s user 0.002s sys 0.000s
I20251026 02:56:43.270920 2378 raft_consensus.cc:359] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:43.271076 2378 raft_consensus.cc:385] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.271132 2378 raft_consensus.cc:740] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.271371 2378 consensus_queue.cc:260] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:43.271695 2027 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:43.278398 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e45600a92f8e46b096badd700779856d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.254040 2377 raft_consensus.cc:2272] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:43.255450 2194 leader_election.cc:343] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:43.279156 2194 leader_election.cc:343] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.279369 3769 raft_consensus.cc:2804] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:43.279433 3769 raft_consensus.cc:493] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:43.279505 3769 raft_consensus.cc:3060] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.279507 2378 ts_tablet_manager.cc:1434] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251026 02:56:43.279801 2378 tablet_bootstrap.cc:492] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:43.281145 2378 tablet_bootstrap.cc:654] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.282066 3769 raft_consensus.cc:515] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.283285 3769 leader_election.cc:290] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:43.283851 2377 ts_tablet_manager.cc:1916] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:43.266867 2381 tablet_bootstrap.cc:492] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:43.260223 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 868dbbfce40c4dd6b36cc220966b1986 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.288812 2381 ts_tablet_manager.cc:1403] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.178s user 0.003s sys 0.000s
I20251026 02:56:43.289186 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 41b7b0c36ba748f0a1565892dbe98a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.289209 1946 tablet_service.cc:1505] Processing CreateTablet for tablet fe27c85771964e4ca0ea747bbb4826df (DEFAULT_TABLE table=default.Table_b17021b7e5d845748cfd8d145b9721ce [id=fca416ada06b470683bd8d22872b6b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.289635 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe27c85771964e4ca0ea747bbb4826df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.261125 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b9dc0825e0844f68d5afcc201db737c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.289953 2381 raft_consensus.cc:359] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.290191 2381 raft_consensus.cc:385] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.290225 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:43.290285 2381 raft_consensus.cc:740] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
W20251026 02:56:43.290743 2194 leader_election.cc:343] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.291213 2381 consensus_queue.cc:260] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.292547 2381 ts_tablet_manager.cc:1434] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:43.292840 2381 tablet_bootstrap.cc:492] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:43.287792 2053 leader_election.cc:304] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.294941 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fb2ff87cf0248a0a4a77f2733ef867b with delete_type TABLET_DATA_DELETED (replaced by tablet ce11081bf85f44e8bf1f92ce0f4bf152) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.296314 2381 tablet_bootstrap.cc:654] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.278309 2376 rpcz_store.cc:276] 1026 02:56:38.693948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.695244 (+ 1296us) service_pool.cc:225] Handling call
1026 02:56:43.164806 (+4469562us) ts_tablet_manager.cc:1151] Deleting tablet efd751ee4b1d433198dcd00c081f648d
1026 02:56:43.164812 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.169424 (+ 4612us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.176282 (+ 6858us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.250906 (+ 74624us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4469300}
I20251026 02:56:43.298413 3843 raft_consensus.cc:2749] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.298626 3843 consensus_queue.cc:1048] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:43.299280 2053 leader_election.cc:343] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.300984 2377 ts_tablet_manager.cc:1929] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:43.301457 2377 log.cc:1199] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/be1c0948c0ae4563b7cb644521914cb9
I20251026 02:56:43.301463 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 752) took 4294 ms. Trace:
I20251026 02:56:43.301549 2376 rpcz_store.cc:276] 1026 02:56:39.006633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.006914 (+ 281us) service_pool.cc:225] Handling call
1026 02:56:43.301331 (+4294417us) ts_tablet_manager.cc:1151] Deleting tablet 46547883cfe041f3954aa4128fba1d8c
1026 02:56:43.301341 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.301444 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4294190}
I20251026 02:56:43.302284 2377 ts_tablet_manager.cc:1950] T be1c0948c0ae4563b7cb644521914cb9 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:43.302619 2053 leader_election.cc:304] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.303565 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 035724949fe142ea93cb998f8fdc4344
I20251026 02:56:43.305145 3843 raft_consensus.cc:2749] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:43.305603 2053 leader_election.cc:343] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.306310 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 753) took 4299 ms. Trace:
I20251026 02:56:43.306571 2053 leader_election.cc:304] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:43.307111 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9bf547b01394be9b126de89c501193b consensus state has no leader
I20251026 02:56:43.307770 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_063e963dcfc24cd78d8072a664472185"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.308326 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_063e963dcfc24cd78d8072a664472185 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.308501 3843 raft_consensus.cc:2749] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.308164 2376 rpcz_store.cc:276] 1026 02:56:39.006862 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.007172 (+ 310us) service_pool.cc:225] Handling call
1026 02:56:43.306155 (+4298983us) ts_tablet_manager.cc:1151] Deleting tablet 02861f399f4d488586f279c01eebcdd1
1026 02:56:43.306165 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.306274 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4298788}
I20251026 02:56:43.309700 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c02fc4c8f1994cabaee4a6aaf1874bdd" } new_table_name: "default.Table_38f23e9599964426a78e8e11b9fa08bb" modify_external_catalogs: true
I20251026 02:56:43.310074 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d87f21f83121454ab8069f24d8b147ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.310570 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d87f21f83121454ab8069f24d8b147ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.299986 2097 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a3f3fc3c7a04d05bae28472251f3acc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.300395 2094 tablet_service.cc:1505] Processing CreateTablet for tablet fe27c85771964e4ca0ea747bbb4826df (DEFAULT_TABLE table=default.Table_b17021b7e5d845748cfd8d145b9721ce [id=fca416ada06b470683bd8d22872b6b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.312055 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe27c85771964e4ca0ea747bbb4826df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.312582 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 6d06944f71fd4e42a6ea46969c86e9f1 (DEFAULT_TABLE table=default.Table_217ad44ad25140129bf8e39c88b3b593 [id=26a5eae9586f4d00b5ebaf9eb3462272]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.313601 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d06944f71fd4e42a6ea46969c86e9f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.313971 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet efd751ee4b1d433198dcd00c081f648d (table default.Table_c2ef795e86444f9f85bb7d0bcca51240 [id=20d9432d9f6443099d7fc0acb952a839]) successfully deleted
W20251026 02:56:43.314638 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 46547883cfe041f3954aa4128fba1d8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46547883cfe041f3954aa4128fba1d8c already in progress: creating tablet
I20251026 02:56:43.279158 2195 leader_election.cc:304] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.315624 3769 raft_consensus.cc:2804] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:43.315722 3769 raft_consensus.cc:493] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:43.315795 3769 raft_consensus.cc:3060] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.318451 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e38f282e8414b7db2a6d5f4a8933e84" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:43.313202 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 035724949fe142ea93cb998f8fdc4344 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:43.268994 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 1104345f80d6418fa907a8a26924c281 (DEFAULT_TABLE table=default.Table_c985fc44cad2462f8e82eadf887551c8 [id=09dc650467064e92a056f6344fda1c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.272941 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 712) took 6754 ms. Trace:
I20251026 02:56:43.304405 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 6d06944f71fd4e42a6ea46969c86e9f1 (DEFAULT_TABLE table=default.Table_217ad44ad25140129bf8e39c88b3b593 [id=26a5eae9586f4d00b5ebaf9eb3462272]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.319008 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1104345f80d6418fa907a8a26924c281. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.319245 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d06944f71fd4e42a6ea46969c86e9f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.319232 3769 raft_consensus.cc:515] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.319891 3769 leader_election.cc:290] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:43.318920 2372 rpcz_store.cc:276] 1026 02:56:36.518949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.519191 (+ 242us) service_pool.cc:225] Handling call
1026 02:56:43.203391 (+6684200us) ts_tablet_manager.cc:1151] Deleting tablet 1c740ab6481e4481b95878c3a185129d
1026 02:56:43.203403 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.272916 (+ 69513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6679206}
W20251026 02:56:43.320472 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 1c740ab6481e4481b95878c3a185129d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c740ab6481e4481b95878c3a185129d already in progress: creating tablet
I20251026 02:56:43.320859 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 713) took 6801 ms. Trace:
I20251026 02:56:43.320941 2372 rpcz_store.cc:276] 1026 02:56:36.519114 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.519407 (+ 293us) service_pool.cc:225] Handling call
1026 02:56:43.320742 (+6801335us) ts_tablet_manager.cc:1151] Deleting tablet 64415f5588db4df0a94676b650fb3631
1026 02:56:43.320752 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.320842 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6796470}
W20251026 02:56:43.321359 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 64415f5588db4df0a94676b650fb3631 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64415f5588db4df0a94676b650fb3631 already in progress: creating tablet
I20251026 02:56:43.321743 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 716) took 6712 ms. Trace:
I20251026 02:56:43.321836 2372 rpcz_store.cc:276] 1026 02:56:36.609626 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.614013 (+ 4387us) service_pool.cc:225] Handling call
1026 02:56:43.321636 (+6707623us) ts_tablet_manager.cc:1151] Deleting tablet c13d89fc829049428363ede24c350ee5
1026 02:56:43.321646 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.321726 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6707208}
I20251026 02:56:43.322184 2195 leader_election.cc:304] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:43.322258 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet c13d89fc829049428363ede24c350ee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c13d89fc829049428363ede24c350ee5 already in progress: creating tablet
I20251026 02:56:43.322677 3769 raft_consensus.cc:2804] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:43.322791 3769 raft_consensus.cc:493] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:43.322913 3769 raft_consensus.cc:3060] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.326574 3769 raft_consensus.cc:515] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:43.327008 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 02861f399f4d488586f279c01eebcdd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02861f399f4d488586f279c01eebcdd1 already in progress: creating tablet
I20251026 02:56:43.325914 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec13c5386abe414ba25589e03d5b850a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:43.327783 2194 leader_election.cc:343] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.327839 2372 tablet_replica.cc:333] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:43.327970 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21b936fa4f6a4ff6af321971a30cccbe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:43.328028 2372 raft_consensus.cc:2243] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:43.328197 2372 raft_consensus.cc:2272] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.328542 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 754) took 4306 ms. Trace:
W20251026 02:56:43.328534 2194 leader_election.cc:343] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.328820 3769 leader_election.cc:290] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:43.331064 2195 leader_election.cc:304] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.333774 2381 tablet_bootstrap.cc:492] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:43.334002 2381 ts_tablet_manager.cc:1403] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20251026 02:56:43.334992 2381 raft_consensus.cc:359] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.335280 2381 raft_consensus.cc:385] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.335412 2381 raft_consensus.cc:740] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.335889 2381 consensus_queue.cc:260] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.337244 2381 ts_tablet_manager.cc:1434] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 02:56:43.340051 2381 tablet_bootstrap.cc:492] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:43.342823 2381 tablet_bootstrap.cc:654] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.332381 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2d6691bb4cf4a8292b0c082e54a91a5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:43.340626 2376 rpcz_store.cc:276] 1026 02:56:39.022072 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.036653 (+ 14581us) service_pool.cc:225] Handling call
1026 02:56:43.328431 (+4291778us) ts_tablet_manager.cc:1151] Deleting tablet 3e424a9c89e64b1c98ada17fa5c31e9e
1026 02:56:43.328444 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.328526 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4291390}
I20251026 02:56:43.344801 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 770) took 4040 ms. Trace:
I20251026 02:56:43.344934 2376 rpcz_store.cc:276] 1026 02:56:39.304571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.304615 (+ 44us) service_pool.cc:225] Handling call
1026 02:56:43.344679 (+4040064us) ts_tablet_manager.cc:1151] Deleting tablet b819b43894574dc9ad52b4081e46628e
1026 02:56:43.344688 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.344783 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3941918}
I20251026 02:56:43.345396 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 771) took 4009 ms. Trace:
I20251026 02:56:43.345495 2376 rpcz_store.cc:276] 1026 02:56:39.335911 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.335965 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:43.345308 (+4009343us) ts_tablet_manager.cc:1151] Deleting tablet 7053d454def04264a956aa75ceaad5c9
1026 02:56:43.345316 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.345384 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3941332}
I20251026 02:56:43.345835 2376 tablet_replica.cc:333] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:43.349110 3622 raft_consensus.cc:2804] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:43.349220 3622 raft_consensus.cc:493] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:43.349270 3622 raft_consensus.cc:3060] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.351840 3622 raft_consensus.cc:515] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.352991 3622 leader_election.cc:290] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:43.353914 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 743) took 5592 ms. Trace:
I20251026 02:56:43.353981 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe48472efa444bf580e4074c96a9de92" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:43.354508 2194 leader_election.cc:343] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.354063 2377 rpcz_store.cc:276] 1026 02:56:37.761165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.763823 (+ 2658us) service_pool.cc:225] Handling call
1026 02:56:43.242382 (+5478559us) ts_tablet_manager.cc:1151] Deleting tablet be1c0948c0ae4563b7cb644521914cb9
1026 02:56:43.242395 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.286928 (+ 44533us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.300801 (+ 13873us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.353896 (+ 53095us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10502,"tablet-delete.queue_time_us":5474450}
I20251026 02:56:43.355000 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 749) took 5559 ms. Trace:
I20251026 02:56:43.355142 2377 rpcz_store.cc:276] 1026 02:56:37.795272 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.795406 (+ 134us) service_pool.cc:225] Handling call
1026 02:56:43.354894 (+5559488us) ts_tablet_manager.cc:1151] Deleting tablet 6979455632e54d2081065954771f34d3
1026 02:56:43.354904 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.354987 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5559161}
I20251026 02:56:43.355664 2377 tablet_replica.cc:333] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:43.356086 2377 raft_consensus.cc:2243] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:43.356721 2377 raft_consensus.cc:2272] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.357806 2377 ts_tablet_manager.cc:1916] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:43.357826 2372 ts_tablet_manager.cc:1916] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:43.332381 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:43.360530 2053 leader_election.cc:343] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.340441 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 249c05690d8f4811904e455e1d078cf6 (DEFAULT_TABLE table=default.Table_645745dc5d2840adaede1a3708978a74 [id=8c650326182846d4bdba4b6e8b1e87e2]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:43.360776 2053 leader_election.cc:343] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.360940 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 249c05690d8f4811904e455e1d078cf6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:43.360973 2053 leader_election.cc:343] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:43.361164 2053 leader_election.cc:343] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.364795 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 5a00bc04b3654f90af342986751344e6 (table default.Table_164cb1784f4645e499339f7163fdfc2b [id=91b4b1a8162441299c680e7c202a96d9]). Expected version 2 got 0
W20251026 02:56:43.366590 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3e424a9c89e64b1c98ada17fa5c31e9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e424a9c89e64b1c98ada17fa5c31e9e already in progress: creating tablet
W20251026 02:56:43.366854 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b819b43894574dc9ad52b4081e46628e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b819b43894574dc9ad52b4081e46628e already in progress: creating tablet
W20251026 02:56:43.367094 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7053d454def04264a956aa75ceaad5c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7053d454def04264a956aa75ceaad5c9 already in progress: creating tablet
W20251026 02:56:43.368479 2316 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.DeleteTable -> {remote=127.1.150.62:44525, user_credentials={real_user=slave}} blocked reactor thread for 147128us
I20251026 02:56:43.373744 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 2c6d0eb08b9142728dd5b1a6d7fea823 (DEFAULT_TABLE table=default.Table_b3f9a7b483b84dd0a53864640ba2b8b1 [id=ae4e52c75a9f4412a7b2ef8a270423a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.374233 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6d0eb08b9142728dd5b1a6d7fea823. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.374934 2085 tablet_service.cc:1505] Processing CreateTablet for tablet a528e0a545274c7490a1614b05a41766 (DEFAULT_TABLE table=default.Table_9894f9db0eaf46a9b4b04db096784cc6 [id=558a1e9d29f04c7a9b681aba88455189]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.375252 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a528e0a545274c7490a1614b05a41766. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.381901 1660 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_7eb7fd8131cc4bce81b6cc09c5062fc0 [id=5cbfb8198a8947b3abd1c0801e92f100] alter complete (version 1)
I20251026 02:56:43.383369 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:43.383616 2120 raft_consensus.cc:2393] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 696476265495492790739ec126a08539 in current term 1: Already voted for candidate 0b6b0a6c4e144982b67980118cddba48 in this term.
I20251026 02:56:43.384341 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec13c5386abe414ba25589e03d5b850a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:43.384538 2120 raft_consensus.cc:3060] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.386937 1660 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet e9cd10092aac40068a1c7f85e514fd37 (table default.Table_fe09be2527a840b09e3e64c27965845d [id=de5272bcc29644208ddd95bdec21254e]). Expected version 1 got 0
I20251026 02:56:43.388090 2120 raft_consensus.cc:2468] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:43.389463 2376 raft_consensus.cc:2243] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:43.389680 2376 raft_consensus.cc:2272] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.390281 2376 ts_tablet_manager.cc:1916] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:43.390745 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe48472efa444bf580e4074c96a9de92" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:43.390928 2117 raft_consensus.cc:3060] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.392956 2120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21b936fa4f6a4ff6af321971a30cccbe" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:43.393108 2120 raft_consensus.cc:3060] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.393276 1962 tablet_service.cc:1505] Processing CreateTablet for tablet a528e0a545274c7490a1614b05a41766 (DEFAULT_TABLE table=default.Table_9894f9db0eaf46a9b4b04db096784cc6 [id=558a1e9d29f04c7a9b681aba88455189]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.393648 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a528e0a545274c7490a1614b05a41766. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.393877 2117 raft_consensus.cc:2468] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:43.396807 2083 tablet_service.cc:1552] Processing DeleteTablet for tablet 035724949fe142ea93cb998f8fdc4344 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.400447 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 0245af5a2dcb4f569b62307965895c1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.401077 1975 raft_consensus.cc:3060] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.402591 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe48472efa444bf580e4074c96a9de92" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.402808 2259 raft_consensus.cc:2393] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b6b0a6c4e144982b67980118cddba48 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:43.402817 2120 raft_consensus.cc:2468] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:43.403227 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e38f282e8414b7db2a6d5f4a8933e84" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:43.403395 2259 raft_consensus.cc:3060] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:43.406260 2194 consensus_peers.cc:597] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:43.406241 2259 raft_consensus.cc:2468] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:43.406683 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2d6691bb4cf4a8292b0c082e54a91a5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.407204 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:43.407405 2259 raft_consensus.cc:2393] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b6b0a6c4e144982b67980118cddba48 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:43.409533 2375 tablet_bootstrap.cc:492] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:43.410367 2375 ts_tablet_manager.cc:1403] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.247s user 0.000s sys 0.004s
I20251026 02:56:43.411299 2375 raft_consensus.cc:359] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.412438 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 30c619ad37364747b6ebf97a73ef45a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:43.412499 2375 raft_consensus.cc:385] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.412719 2375 raft_consensus.cc:740] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.413259 2378 tablet_bootstrap.cc:492] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:43.413441 2378 ts_tablet_manager.cc:1403] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.134s user 0.030s sys 0.029s
I20251026 02:56:43.413566 2375 consensus_queue.cc:260] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.414242 2378 raft_consensus.cc:359] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.414413 2378 raft_consensus.cc:385] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.414520 2378 raft_consensus.cc:740] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.414783 2378 consensus_queue.cc:260] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.415326 2378 ts_tablet_manager.cc:1434] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:43.415633 2378 tablet_bootstrap.cc:492] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:43.417114 2378 tablet_bootstrap.cc:654] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.419016 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 2c6d0eb08b9142728dd5b1a6d7fea823 (DEFAULT_TABLE table=default.Table_b3f9a7b483b84dd0a53864640ba2b8b1 [id=ae4e52c75a9f4412a7b2ef8a270423a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.419452 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6d0eb08b9142728dd5b1a6d7fea823. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:43.432865 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1a1b8c440061429caea0f0803f110742 consensus state has no leader
W20251026 02:56:43.433899 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0afb6f04f5ac463780fe8f522bf2fe23 consensus state has no leader
I20251026 02:56:43.434161 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet bcd5621aa3754733b22508be89f080db
I20251026 02:56:43.434654 1662 catalog_manager.cc:6719] ReplaceTablet: tablet bcd5621aa3754733b22508be89f080db deleted and replaced by tablet c34406d36c6e48dc973584bd653a9e42
I20251026 02:56:43.445389 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 6403483a7e0f4676a8a88ce55beeae80 (DEFAULT_TABLE table=default.Table_de99eb535be548db8335e7fbcfe8adb7 [id=ea3b1f4c26e842b28c9bb9b9341439d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.445806 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6403483a7e0f4676a8a88ce55beeae80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.445842 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 868dbbfce40c4dd6b36cc220966b1986 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.446082 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b9dc0825e0844f68d5afcc201db737c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.446309 2233 tablet_service.cc:1505] Processing CreateTablet for tablet e4c149e8b7f9420c8dba56da2e21f3f9 (DEFAULT_TABLE table=default.Table_0d7b9434a0494a618c25ceebd7f65de8 [id=bb7b13116ad94fa9bd15a1cdb15d207b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.446480 2230 tablet_service.cc:1505] Processing CreateTablet for tablet a528e0a545274c7490a1614b05a41766 (DEFAULT_TABLE table=default.Table_9894f9db0eaf46a9b4b04db096784cc6 [id=558a1e9d29f04c7a9b681aba88455189]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.446594 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4c149e8b7f9420c8dba56da2e21f3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.446748 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a528e0a545274c7490a1614b05a41766. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:43.447358 2195 consensus_peers.cc:597] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a3491b303f94fe2a2c838297f73d464. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:43.447929 2195 leader_election.cc:304] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.448136 3622 raft_consensus.cc:2749] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.448297 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 19918cb3b7d0450f9de17026f8ea3295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.448501 2195 leader_election.cc:304] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.448601 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 1104345f80d6418fa907a8a26924c281 (DEFAULT_TABLE table=default.Table_c985fc44cad2462f8e82eadf887551c8 [id=09dc650467064e92a056f6344fda1c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.448683 3622 raft_consensus.cc:2804] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:43.448763 3622 raft_consensus.cc:697] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:43.448869 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1104345f80d6418fa907a8a26924c281. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.448973 3622 consensus_queue.cc:237] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.449759 2225 tablet_service.cc:1505] Processing CreateTablet for tablet fe27c85771964e4ca0ea747bbb4826df (DEFAULT_TABLE table=default.Table_b17021b7e5d845748cfd8d145b9721ce [id=fca416ada06b470683bd8d22872b6b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.450035 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe27c85771964e4ca0ea747bbb4826df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.452386 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 035724949fe142ea93cb998f8fdc4344 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.452498 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 12e10fd463cd43c7abb22ef428c0aa37 (DEFAULT_TABLE table=default.Table_cbf5e72aa3ea4bcea06df860ce74bb42 [id=b31c19a4727a4647a8bab645f5c784f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.452981 2229 tablet_service.cc:1505] Processing CreateTablet for tablet e45600a92f8e46b096badd700779856d (DEFAULT_TABLE table=default.Table_81d15a2f4d424910a51421472c159f0d [id=0f4a4be6bea7411aa2e13984864e827c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.453158 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 2c6d0eb08b9142728dd5b1a6d7fea823 (DEFAULT_TABLE table=default.Table_b3f9a7b483b84dd0a53864640ba2b8b1 [id=ae4e52c75a9f4412a7b2ef8a270423a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.446309 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 249c05690d8f4811904e455e1d078cf6 (DEFAULT_TABLE table=default.Table_645745dc5d2840adaede1a3708978a74 [id=8c650326182846d4bdba4b6e8b1e87e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.453522 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 4984a3aea6a245ffba7f393b47add7d0 (DEFAULT_TABLE table=default.Table_efc39d27d9d5438ea04cef669a110c8a [id=2494ffc89efd452d9778517b731a3dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.454031 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 716007b9d96e43699033e8a5879afcac (DEFAULT_TABLE table=default.Table_52557bc43e124635b11e4e542e6f62df [id=8f7c804d94984a0b9451c3e1e86d79da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.454381 2227 tablet_service.cc:1552] Processing DeleteTablet for tablet 0245af5a2dcb4f569b62307965895c1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.454581 2375 ts_tablet_manager.cc:1434] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.044s user 0.001s sys 0.001s
I20251026 02:56:43.454942 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 249c05690d8f4811904e455e1d078cf6 (DEFAULT_TABLE table=default.Table_645745dc5d2840adaede1a3708978a74 [id=8c650326182846d4bdba4b6e8b1e87e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.456020 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 6d06944f71fd4e42a6ea46969c86e9f1 (DEFAULT_TABLE table=default.Table_217ad44ad25140129bf8e39c88b3b593 [id=26a5eae9586f4d00b5ebaf9eb3462272]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:43.457646 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 249c05690d8f4811904e455e1d078cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.458204 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e45600a92f8e46b096badd700779856d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.458616 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 716007b9d96e43699033e8a5879afcac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.458998 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e10fd463cd43c7abb22ef428c0aa37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.459386 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4984a3aea6a245ffba7f393b47add7d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.459741 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6d0eb08b9142728dd5b1a6d7fea823. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.460095 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d06944f71fd4e42a6ea46969c86e9f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.466688 2377 ts_tablet_manager.cc:1929] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:43.466800 2377 log.cc:1199] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/741792cf96164695b03de3fb86b12602
I20251026 02:56:43.467106 2377 ts_tablet_manager.cc:1950] T 741792cf96164695b03de3fb86b12602 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:43.471886 2195 leader_election.cc:304] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.472416 3537 raft_consensus.cc:2804] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:43.472522 3537 raft_consensus.cc:697] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:43.472716 3537 consensus_queue.cc:237] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.482225 2053 leader_election.cc:304] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.482613 3846 raft_consensus.cc:2749] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.483047 2053 leader_election.cc:304] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.483372 3846 raft_consensus.cc:2804] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:43.483487 3846 raft_consensus.cc:697] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:43.486088 3372 raft_consensus.cc:493] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.486238 3372 raft_consensus.cc:515] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.486938 3372 leader_election.cc:290] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:43.488408 2053 leader_election.cc:343] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.488551 2053 leader_election.cc:304] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.488809 3842 raft_consensus.cc:2749] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.492733 3769 consensus_queue.cc:1048] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:43.493815 3866 tablet.cc:1722] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:43.494135 2195 leader_election.cc:304] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.494961 3769 raft_consensus.cc:2804] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:43.495095 3769 raft_consensus.cc:697] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:43.483793 3846 consensus_queue.cc:237] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.495299 3769 consensus_queue.cc:237] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.501842 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet cfc3aed64ad0439cb22fd6815c973d87 (table default.Table_34881d9847d5463297830bf6cc4ed1e6 [id=a484f36091f24fd2b8bc2af4f53eb450]) successfully deleted
W20251026 02:56:43.502079 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 83355.6us
W20251026 02:56:43.502380 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7f5cd37c87f34a40bf05829af6de0473 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f5cd37c87f34a40bf05829af6de0473 already in progress: creating tablet
I20251026 02:56:43.507408 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet be1c0948c0ae4563b7cb644521914cb9 (table default.Table_d197f56404544c8cb996d53edb9559a4 [id=777761bfdc67409bb98be7cf82bfa547]) successfully deleted
W20251026 02:56:43.508137 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 6979455632e54d2081065954771f34d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6979455632e54d2081065954771f34d3 already in progress: creating tablet
I20251026 02:56:43.512976 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6f9ae327aa4401baf66a737eaa93cf6" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.515566 2308 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:43.496125 2372 ts_tablet_manager.cc:1929] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:43.500396 2053 leader_election.cc:304] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.523481 3842 raft_consensus.cc:2749] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.523638 2381 tablet_bootstrap.cc:492] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:43.523872 2381 ts_tablet_manager.cc:1403] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.184s user 0.004s sys 0.000s
I20251026 02:56:43.525524 2381 raft_consensus.cc:359] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.525888 2381 raft_consensus.cc:385] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.526043 2381 raft_consensus.cc:740] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.526420 2381 consensus_queue.cc:260] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.528446 2378 tablet_bootstrap.cc:492] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:43.528771 2378 ts_tablet_manager.cc:1403] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.113s user 0.011s sys 0.013s
I20251026 02:56:43.529956 2378 raft_consensus.cc:359] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.530287 2378 raft_consensus.cc:385] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.530442 2378 raft_consensus.cc:740] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.530918 2378 consensus_queue.cc:260] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.531984 2372 log.cc:1199] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/f719c720cc474181a63786a3e248ab2c
I20251026 02:56:43.532573 2372 ts_tablet_manager.cc:1950] T f719c720cc474181a63786a3e248ab2c P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:43.540622 3877 raft_consensus.cc:493] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.541293 3877 raft_consensus.cc:515] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.542600 3877 leader_election.cc:290] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:43.542572 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f437d8b5f5784e8fbec10d0700e90348" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:43.543146 2194 leader_election.cc:343] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.545548 2375 tablet_bootstrap.cc:492] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:43.545863 1916 leader_election.cc:343] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: a6f9ae327aa4401baf66a737eaa93cf6
I20251026 02:56:43.546011 1916 leader_election.cc:304] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:43.546428 3372 raft_consensus.cc:2749] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.546676 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet bcd5621aa3754733b22508be89f080db with delete_type TABLET_DATA_DELETED (replaced by tablet c34406d36c6e48dc973584bd653a9e42) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:43.546897 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 249c05690d8f4811904e455e1d078cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:43.548053 2381 ts_tablet_manager.cc:1434] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20251026 02:56:43.548250 2378 ts_tablet_manager.cc:1434] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251026 02:56:43.549024 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f437d8b5f5784e8fbec10d0700e90348" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:43.549212 2109 raft_consensus.cc:2468] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:43.549999 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 750) took 5754 ms. Trace:
I20251026 02:56:43.550079 2377 rpcz_store.cc:276] 1026 02:56:37.795367 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.831949 (+ 36582us) service_pool.cc:225] Handling call
1026 02:56:43.355573 (+5523624us) ts_tablet_manager.cc:1151] Deleting tablet 741792cf96164695b03de3fb86b12602
1026 02:56:43.355605 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.463778 (+108173us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.466603 (+ 2825us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.549977 (+ 83374us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82090,"tablet-delete.queue_time_us":5523380}
I20251026 02:56:43.550482 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 751) took 5718 ms. Trace:
I20251026 02:56:43.550552 2377 rpcz_store.cc:276] 1026 02:56:37.832364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.832403 (+ 39us) service_pool.cc:225] Handling call
1026 02:56:43.550342 (+5717939us) ts_tablet_manager.cc:1151] Deleting tablet d7313a6396c541d7ac6d8ca2e219af90
1026 02:56:43.550351 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.550467 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5717801}
I20251026 02:56:43.550865 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 756) took 5714 ms. Trace:
I20251026 02:56:43.550922 2377 rpcz_store.cc:276] 1026 02:56:37.836422 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.836482 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:43.550778 (+5714296us) ts_tablet_manager.cc:1151] Deleting tablet 9a5670a9e10c4a9db64881e7e14e839e
1026 02:56:43.550786 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.550853 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5714121}
I20251026 02:56:43.551105 2377 tablet_replica.cc:333] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:43.551262 2377 raft_consensus.cc:2243] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:43.551430 2377 raft_consensus.cc:2272] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.551954 2377 ts_tablet_manager.cc:1916] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:43.552661 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet bcd5621aa3754733b22508be89f080db with delete_type TABLET_DATA_DELETED (replaced by tablet c34406d36c6e48dc973584bd653a9e42) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.559664 2376 ts_tablet_manager.cc:1929] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251026 02:56:43.561561 2381 tablet_bootstrap.cc:492] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:43.562108 2378 tablet_bootstrap.cc:492] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:43.562361 2376 log.cc:1199] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/cfc3aed64ad0439cb22fd6815c973d87
I20251026 02:56:43.563160 3622 consensus_queue.cc:1048] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:43.566242 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:43.566780 2375 tablet_bootstrap.cc:654] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.567885 2195 leader_election.cc:304] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.565608 2376 ts_tablet_manager.cc:1950] T cfc3aed64ad0439cb22fd6815c973d87 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:43.569355 3239 raft_consensus.cc:2804] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:43.569458 3239 raft_consensus.cc:493] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:43.569525 3239 raft_consensus.cc:3060] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.569092 1975 raft_consensus.cc:1275] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 02:56:43.570429 2381 tablet_bootstrap.cc:654] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.570659 3769 consensus_queue.cc:1048] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:43.570977 2378 tablet_bootstrap.cc:654] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.572599 3239 raft_consensus.cc:515] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.573221 3841 raft_consensus.cc:493] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.573370 3841 raft_consensus.cc:515] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.573676 3239 leader_election.cc:290] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:43.573961 3841 leader_election.cc:290] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.574196 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f437d8b5f5784e8fbec10d0700e90348" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:43.575358 2194 leader_election.cc:343] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.575821 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0bacd15b41149b5856d20068711b432" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:43.570102 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 772) took 3986 ms. Trace:
I20251026 02:56:43.576399 2376 rpcz_store.cc:276] 1026 02:56:39.583501 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.583586 (+ 85us) service_pool.cc:225] Handling call
1026 02:56:43.345800 (+3762214us) ts_tablet_manager.cc:1151] Deleting tablet cfc3aed64ad0439cb22fd6815c973d87
1026 02:56:43.345807 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.439534 (+ 93727us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.559515 (+119981us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.570085 (+ 10570us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43389,"tablet-delete.queue_time_us":3761979}
I20251026 02:56:43.576781 2376 tablet_replica.cc:333] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:43.577137 2376 raft_consensus.cc:2243] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:43.577347 2376 raft_consensus.cc:2272] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.577930 2376 ts_tablet_manager.cc:1916] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:43.579362 2053 leader_election.cc:343] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:43.582351 1649 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 4be3e9d5b07a48999497ed54a3d035fe (table default.Table_fc6775059a3d41bdac1af3af53ffd921 [id=c89fbfa5c47d4bcca61df44679baddd9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4be3e9d5b07a48999497ed54a3d035fe consensus state has no leader
W20251026 02:56:43.585125 2194 consensus_peers.cc:597] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:43.586947 3496 log.cc:927] Time spent T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Append to log took a long time: real 0.059s user 0.003s sys 0.000s
I20251026 02:56:43.590392 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 717) took 6966 ms. Trace:
I20251026 02:56:43.591104 2372 rpcz_store.cc:276] 1026 02:56:36.624085 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.624317 (+ 232us) service_pool.cc:225] Handling call
1026 02:56:43.327788 (+6703471us) ts_tablet_manager.cc:1151] Deleting tablet f719c720cc474181a63786a3e248ab2c
1026 02:56:43.327799 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.428424 (+100625us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.496011 (+ 67587us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.590146 (+ 94135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6703214}
I20251026 02:56:43.592494 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 718) took 6873 ms. Trace:
I20251026 02:56:43.593418 2372 rpcz_store.cc:276] 1026 02:56:36.718885 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.721656 (+ 2771us) service_pool.cc:225] Handling call
1026 02:56:43.592350 (+6870694us) ts_tablet_manager.cc:1151] Deleting tablet 58d60bf2a9e546e6a9474a5af6d5273a
1026 02:56:43.592362 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.592474 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6870482}
I20251026 02:56:43.594565 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 719) took 6875 ms. Trace:
I20251026 02:56:43.595242 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f437d8b5f5784e8fbec10d0700e90348" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:43.595590 2109 raft_consensus.cc:3060] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.596740 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0bacd15b41149b5856d20068711b432" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:43.598260 2194 consensus_peers.cc:597] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:43.595286 2372 rpcz_store.cc:276] 1026 02:56:36.719002 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.721900 (+ 2898us) service_pool.cc:225] Handling call
1026 02:56:43.594472 (+6872572us) ts_tablet_manager.cc:1151] Deleting tablet 63a63b796191465aab9a63d3ffb0554d
1026 02:56:43.594484 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.594550 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6872447}
I20251026 02:56:43.599786 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 720) took 6879 ms. Trace:
I20251026 02:56:43.599862 2372 rpcz_store.cc:276] 1026 02:56:36.720169 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.722050 (+ 1881us) service_pool.cc:225] Handling call
1026 02:56:43.599675 (+6877625us) ts_tablet_manager.cc:1151] Deleting tablet 244c03a9293242a48077bdaf90a47865
1026 02:56:43.599685 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.599769 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6877511}
I20251026 02:56:43.601954 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 722) took 6627 ms. Trace:
I20251026 02:56:43.602535 2372 rpcz_store.cc:276] 1026 02:56:36.974249 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.974306 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:43.601842 (+6627536us) ts_tablet_manager.cc:1151] Deleting tablet ded06ba1f80c4433ae78687e5990edce
1026 02:56:43.601852 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.601936 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6627306}
I20251026 02:56:43.602988 2372 tablet_replica.cc:333] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:43.603116 2109 raft_consensus.cc:2468] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:43.603649 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7df0c57231f6477fa64624f1bec15130" } new_table_name: "default.Table_0c1f6e61ec3e4936acccf4e7598580b9" modify_external_catalogs: true
I20251026 02:56:43.604100 2378 tablet_bootstrap.cc:492] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:43.604118 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_439dbab03ccd44c7869b2f12fb6431c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:43.604373 2378 ts_tablet_manager.cc:1403] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.001s
I20251026 02:56:43.604460 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1c36ab4645984dc4b2280de23324521b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.604707 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_439dbab03ccd44c7869b2f12fb6431c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:43.604880 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c36ab4645984dc4b2280de23324521b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.604979 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet cfc3aed64ad0439cb22fd6815c973d87 (table default.Table_34881d9847d5463297830bf6cc4ed1e6 [id=a484f36091f24fd2b8bc2af4f53eb450]) successfully deleted
I20251026 02:56:43.605686 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 741792cf96164695b03de3fb86b12602 (table default.Table_d4304259ac684b8097cdbf5511ab67d7 [id=d64901838ed94621ad11692d9e0aa80e]) successfully deleted
I20251026 02:56:43.605667 2378 raft_consensus.cc:359] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:43.605878 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet d7313a6396c541d7ac6d8ca2e219af90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7313a6396c541d7ac6d8ca2e219af90 already in progress: creating tablet
I20251026 02:56:43.605942 2378 raft_consensus.cc:385] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.605911 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0c717ffb0f0440c3a035bb9717943e08" } new_table_name: "default.Table_431d446f4c7b4a7b85f4386e28da2d3b" modify_external_catalogs: true
W20251026 02:56:43.606057 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 9a5670a9e10c4a9db64881e7e14e839e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a5670a9e10c4a9db64881e7e14e839e already in progress: creating tablet
I20251026 02:56:43.606029 2378 raft_consensus.cc:740] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.606129 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4836734d4c9f4d37b4f223c1147dbc30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.606504 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4836734d4c9f4d37b4f223c1147dbc30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.606388 2378 consensus_queue.cc:260] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.607082 2378 ts_tablet_manager.cc:1434] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:43.607106 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_517fcc0e025a4b8f9ec1a392735037a9" } modify_external_catalogs: true
I20251026 02:56:43.607230 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_517fcc0e025a4b8f9ec1a392735037a9" } modify_external_catalogs: true
I20251026 02:56:43.609161 2381 tablet_bootstrap.cc:492] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:43.609562 2381 ts_tablet_manager.cc:1403] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.006s
I20251026 02:56:43.608147 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b754adc158c44242939aa8d9878827e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.613489 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b754adc158c44242939aa8d9878827e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:43.614737 2053 leader_election.cc:343] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.614866 2053 leader_election.cc:304] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.615489 3841 raft_consensus.cc:2749] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.603374 2372 raft_consensus.cc:2243] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:43.617627 2372 raft_consensus.cc:2272] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.618528 2372 ts_tablet_manager.cc:1916] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:43.619508 2194 consensus_peers.cc:597] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:43.621852 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f80b5c4c4e2d4c14a8a5292a05127bc1 has no consensus state
I20251026 02:56:43.622216 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_063e963dcfc24cd78d8072a664472185" } modify_external_catalogs: true
I20251026 02:56:43.622450 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_063e963dcfc24cd78d8072a664472185" } modify_external_catalogs: true
I20251026 02:56:43.622201 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_afa20b14ea2b4ebab253b18ecca40a69" } modify_external_catalogs: true
I20251026 02:56:43.623716 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_afa20b14ea2b4ebab253b18ecca40a69" } modify_external_catalogs: true
I20251026 02:56:43.625792 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet f437d8b5f5784e8fbec10d0700e90348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.626045 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet aa10cd990a944eab8d9eb66aaf20080c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.627583 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ce756cf85c594b7f97cbbfb345be2856" } modify_external_catalogs: true
I20251026 02:56:43.627732 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ce756cf85c594b7f97cbbfb345be2856" } modify_external_catalogs: true
I20251026 02:56:43.629599 2195 leader_election.cc:304] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.630110 3537 raft_consensus.cc:2804] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:43.630226 3537 raft_consensus.cc:697] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:43.630475 3537 consensus_queue.cc:237] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.632795 2378 tablet_bootstrap.cc:492] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:43.642241 1655 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet ab923e219ff442939b66f1ad0902151c
I20251026 02:56:43.642791 3877 raft_consensus.cc:493] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.642932 3877 raft_consensus.cc:515] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.643503 3877 leader_election.cc:290] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:43.644392 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6288384ccff04bfca7c4b0a988abfc09" } modify_external_catalogs: true
I20251026 02:56:43.644606 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6288384ccff04bfca7c4b0a988abfc09" } modify_external_catalogs: true
I20251026 02:56:43.645394 1657 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet b96393df0bdf443cab4f150835fd4326
I20251026 02:56:43.648021 2378 tablet_bootstrap.cc:654] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.648725 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ea91bad0b400465a98d6aec2534bcabe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.650110 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea91bad0b400465a98d6aec2534bcabe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.651432 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cbf5e72aa3ea4bcea06df860ce74bb42" } new_table_name: "default.Table_e1d771f7ba5b433b91c68d98c6e48cf1" modify_external_catalogs: true
I20251026 02:56:43.652585 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c7ef2f8329744217bb25610a6afb4889"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.653024 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7ef2f8329744217bb25610a6afb4889 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.653838 2372 ts_tablet_manager.cc:1929] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:43.655969 2372 log.cc:1199] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3dbe8002df6c4b64b8ec8260ca6b9cf5
I20251026 02:56:43.656147 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet de8799a5af2b42cc92ccc0195cff98e7 (table default.Table_13667e60be9c46b192cebf20df523e99 [id=7885bd43f8a245b0a3010a63dbd535f5]). Expected version 3 got 0
I20251026 02:56:43.656493 2372 ts_tablet_manager.cc:1950] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:43.658783 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4e1e3945b1094e179d628f897d2241dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.659392 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e1e3945b1094e179d628f897d2241dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.660564 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 724) took 6681 ms. Trace:
I20251026 02:56:43.660717 2372 rpcz_store.cc:276] 1026 02:56:36.979520 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.979574 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:43.602949 (+6623375us) ts_tablet_manager.cc:1151] Deleting tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5
1026 02:56:43.602957 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.650601 (+ 47644us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.653720 (+ 3119us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.660541 (+ 6821us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6623160}
I20251026 02:56:43.661695 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 725) took 6682 ms. Trace:
I20251026 02:56:43.662148 2372 rpcz_store.cc:276] 1026 02:56:36.979658 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.979834 (+ 176us) service_pool.cc:225] Handling call
1026 02:56:43.661584 (+6681750us) ts_tablet_manager.cc:1151] Deleting tablet 0fe0e06a86e241bc91a1db4219fe25bc
1026 02:56:43.661597 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.661675 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6681574}
I20251026 02:56:43.662638 2372 tablet_replica.cc:333] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:43.663290 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 86e483c52e514f1da54335479f59d68e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.663352 2372 raft_consensus.cc:2243] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:43.662752 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa10cd990a944eab8d9eb66aaf20080c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:43.663528 2372 raft_consensus.cc:2272] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.663601 2116 raft_consensus.cc:2468] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:43.664559 2372 ts_tablet_manager.cc:1916] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:43.666266 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8fdadd053cc34aeca0d491eb3200207f
I20251026 02:56:43.670046 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_79c1a661d818462f8b55978d4541f5b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:43.670472 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fdadd053cc34aeca0d491eb3200207f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:43.670666 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79c1a661d818462f8b55978d4541f5b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:43.671928 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 947ad059be4b475f92b5ba208e3dd9e5 consensus state has no leader
I20251026 02:56:43.673831 2375 tablet_bootstrap.cc:492] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:43.673990 2378 tablet_bootstrap.cc:492] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:43.674180 2378 ts_tablet_manager.cc:1403] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.005s
I20251026 02:56:43.674249 2375 ts_tablet_manager.cc:1403] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.130s user 0.007s sys 0.011s
I20251026 02:56:43.675587 2378 raft_consensus.cc:359] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.675678 2195 leader_election.cc:304] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:43.675801 2378 raft_consensus.cc:385] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.675879 2378 raft_consensus.cc:740] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.675892 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fdadd053cc34aeca0d491eb3200207f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.676162 2378 consensus_queue.cc:260] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.676906 2375 raft_consensus.cc:359] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.677320 2375 raft_consensus.cc:385] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.677523 2375 raft_consensus.cc:740] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.678043 2375 consensus_queue.cc:260] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.678947 1663 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 8862cd48a5744c99b32f365cb77fa7c4 (table default.Table_dabd227b99964351870151b7731b8175 [id=604570227c014f66aeae89d93edaaafe]). Expected version 1 got 0
I20251026 02:56:43.630129 2381 raft_consensus.cc:359] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.679714 2381 raft_consensus.cc:385] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.679783 2381 raft_consensus.cc:740] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.679981 2381 consensus_queue.cc:260] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.681244 3888 raft_consensus.cc:2804] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:43.681350 3888 raft_consensus.cc:493] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:43.681432 3888 raft_consensus.cc:3060] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:43.682646 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12e10fd463cd43c7abb22ef428c0aa37 consensus state has no leader
I20251026 02:56:43.685747 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa10cd990a944eab8d9eb66aaf20080c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:43.686348 2194 leader_election.cc:343] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.687785 3372 raft_consensus.cc:493] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.687958 3372 raft_consensus.cc:515] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.688750 3372 leader_election.cc:290] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.689397 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8862cd48a5744c99b32f365cb77fa7c4" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:43.690582 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8862cd48a5744c99b32f365cb77fa7c4" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.691224 1915 leader_election.cc:304] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:43.691748 3372 raft_consensus.cc:2749] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.691627 2375 ts_tablet_manager.cc:1434] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251026 02:56:43.695780 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet b1ca5b8379c94deab1130bfe8bdc87b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:43.695780 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c6e3e47b3e7421e945cfbe671b06abe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:43.697103 2378 ts_tablet_manager.cc:1434] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20251026 02:56:43.702775 2375 tablet_bootstrap.cc:492] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:43.705502 2377 ts_tablet_manager.cc:1929] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:43.705646 2377 log.cc:1199] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/8688a0bf946841deb5d88c931538b129
I20251026 02:56:43.706029 2377 ts_tablet_manager.cc:1950] T 8688a0bf946841deb5d88c931538b129 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:43.707558 2378 tablet_bootstrap.cc:492] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:43.710899 2372 ts_tablet_manager.cc:1929] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:43.712206 2378 tablet_bootstrap.cc:654] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.710999 2375 tablet_bootstrap.cc:654] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.713205 2372 log.cc:1199] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7801f073d5d94076af9299492f86fad2
I20251026 02:56:43.713773 2372 ts_tablet_manager.cc:1950] T 7801f073d5d94076af9299492f86fad2 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:43.714394 2195 consensus_peers.cc:597] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbf2a1ce78154415b23ab6d0efbc1f52. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:43.714954 3841 raft_consensus.cc:493] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.715103 3841 raft_consensus.cc:515] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.715844 3841 leader_election.cc:290] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.716492 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a012d868a78f4324a92db83c81931b9d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:43.717160 2053 leader_election.cc:343] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.718039 2378 tablet_bootstrap.cc:492] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:43.718215 2378 ts_tablet_manager.cc:1403] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20251026 02:56:43.718357 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a012d868a78f4324a92db83c81931b9d" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:43.718780 2053 leader_election.cc:343] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.718878 2053 leader_election.cc:304] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.718950 2378 raft_consensus.cc:359] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.719106 2378 raft_consensus.cc:385] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.719161 2378 raft_consensus.cc:740] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.719278 3841 raft_consensus.cc:2749] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.719393 2378 consensus_queue.cc:260] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.722025 2381 ts_tablet_manager.cc:1434] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.112s user 0.001s sys 0.003s
I20251026 02:56:43.722396 2381 tablet_bootstrap.cc:492] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:43.726001 3372 raft_consensus.cc:493] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.726143 3372 raft_consensus.cc:515] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:43.726789 3372 leader_election.cc:290] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:43.727784 1916 leader_election.cc:343] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 30c619ad37364747b6ebf97a73ef45a2
W20251026 02:56:43.731724 1915 leader_election.cc:343] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 30c619ad37364747b6ebf97a73ef45a2
I20251026 02:56:43.731727 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:43.731907 1915 leader_election.cc:304] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:43.732369 3372 raft_consensus.cc:2749] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:43.736344 3496 log.cc:927] Time spent T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Append to log took a long time: real 0.142s user 0.002s sys 0.000s
I20251026 02:56:43.737695 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 723) took 6758 ms. Trace:
I20251026 02:56:43.737818 2372 rpcz_store.cc:276] 1026 02:56:36.979372 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:36.979848 (+ 476us) service_pool.cc:225] Handling call
1026 02:56:43.662586 (+6682738us) ts_tablet_manager.cc:1151] Deleting tablet 7801f073d5d94076af9299492f86fad2
1026 02:56:43.662597 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.671878 (+ 9281us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.710796 (+ 38918us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.737669 (+ 26873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6682428}
I20251026 02:56:43.738157 2378 ts_tablet_manager.cc:1434] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20251026 02:56:43.738579 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 729) took 6634 ms. Trace:
I20251026 02:56:43.738710 2372 rpcz_store.cc:276] 1026 02:56:37.104532 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.125312 (+ 20780us) service_pool.cc:225] Handling call
1026 02:56:43.738448 (+6613136us) ts_tablet_manager.cc:1151] Deleting tablet 1a36b8a4beca483bb853d27961f5281a
1026 02:56:43.738461 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.738560 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6610944}
I20251026 02:56:43.739234 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 731) took 6608 ms. Trace:
I20251026 02:56:43.739359 2372 rpcz_store.cc:276] 1026 02:56:37.131101 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.132317 (+ 1216us) service_pool.cc:225] Handling call
1026 02:56:43.739135 (+6606818us) ts_tablet_manager.cc:1151] Deleting tablet b68388b2fd014dd1a5f6572796cf53a6
1026 02:56:43.739143 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.739220 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6606280}
I20251026 02:56:43.739802 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 732) took 6604 ms. Trace:
I20251026 02:56:43.739920 2372 rpcz_store.cc:276] 1026 02:56:37.135614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.137796 (+ 2182us) service_pool.cc:225] Handling call
1026 02:56:43.739715 (+6601919us) ts_tablet_manager.cc:1151] Deleting tablet 81775055d98349f696cdab6171ffd58b
1026 02:56:43.739723 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.739789 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6601606}
I20251026 02:56:43.741617 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 730) took 6619 ms. Trace:
I20251026 02:56:43.741626 3888 raft_consensus.cc:515] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.741744 2372 rpcz_store.cc:276] 1026 02:56:37.120739 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.138021 (+ 17282us) service_pool.cc:225] Handling call
1026 02:56:43.740320 (+6602299us) ts_tablet_manager.cc:1151] Deleting tablet 7e1a45412db1449a99252112d8110a17
1026 02:56:43.740329 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.740392 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6601352}
I20251026 02:56:43.742254 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 733) took 6584 ms. Trace:
I20251026 02:56:43.742427 3888 leader_election.cc:290] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:43.742333 2372 rpcz_store.cc:276] 1026 02:56:37.157520 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.157755 (+ 235us) service_pool.cc:225] Handling call
1026 02:56:43.742160 (+6584405us) ts_tablet_manager.cc:1151] Deleting tablet e830f1aa1270420c9b28a0491a1a0960
1026 02:56:43.742168 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.742239 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6584044}
I20251026 02:56:43.742897 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 734) took 6585 ms. Trace:
I20251026 02:56:43.743032 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa10cd990a944eab8d9eb66aaf20080c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:43.742966 2372 rpcz_store.cc:276] 1026 02:56:37.157654 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.157719 (+ 65us) service_pool.cc:225] Handling call
1026 02:56:43.742757 (+6585038us) ts_tablet_manager.cc:1151] Deleting tablet e32a1f8dcad4430eb35d388bfad7d110
1026 02:56:43.742765 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.742882 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6584292}
I20251026 02:56:43.743207 2116 raft_consensus.cc:3060] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.743439 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 741) took 6385 ms. Trace:
I20251026 02:56:43.743508 2372 rpcz_store.cc:276] 1026 02:56:37.357981 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.358054 (+ 73us) service_pool.cc:225] Handling call
1026 02:56:43.743335 (+6385281us) ts_tablet_manager.cc:1151] Deleting tablet b8cad7adae7e43469ceadbedeee53564
1026 02:56:43.743344 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.743426 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6385067}
I20251026 02:56:43.744007 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 742) took 6380 ms. Trace:
I20251026 02:56:43.744114 2372 rpcz_store.cc:276] 1026 02:56:37.363840 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.364044 (+ 204us) service_pool.cc:225] Handling call
1026 02:56:43.743850 (+6379806us) ts_tablet_manager.cc:1151] Deleting tablet ba08469339184b11b9ced9e6bdd63d42
1026 02:56:43.743859 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.743981 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6379551}
I20251026 02:56:43.744630 2372 tablet_replica.cc:333] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:43.744896 2372 raft_consensus.cc:2243] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:43.745144 2372 raft_consensus.cc:2272] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.746004 2372 ts_tablet_manager.cc:1916] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:43.746035 2116 raft_consensus.cc:2468] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:43.747277 2195 leader_election.cc:304] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:43.747897 3239 raft_consensus.cc:2804] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:43.748056 3239 raft_consensus.cc:697] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:43.748389 3239 consensus_queue.cc:237] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.749256 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_79c1a661d818462f8b55978d4541f5b5" } new_table_name: "default.Table_22ed9a263e5942e4b6fdd13731e49e1d" modify_external_catalogs: true
I20251026 02:56:43.751040 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4f8a62d4db7e4215938585f18149e939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:43.751180 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c4af4099c79b4cd9a7e7ef8a0d762dbf" } new_table_name: "default.Table_12cdd920c7d54ce7818f9fa6d5fce036" modify_external_catalogs: true
I20251026 02:56:43.751477 2381 tablet_bootstrap.cc:654] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:43.751729 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f8a62d4db7e4215938585f18149e939 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.752077 2378 tablet_bootstrap.cc:492] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:43.753736 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 757) took 5916 ms. Trace:
I20251026 02:56:43.753904 2377 rpcz_store.cc:276] 1026 02:56:37.836824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.836881 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:43.551073 (+5714192us) ts_tablet_manager.cc:1151] Deleting tablet 8688a0bf946841deb5d88c931538b129
1026 02:56:43.551079 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.667670 (+116591us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.705364 (+ 37694us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.753715 (+ 48351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5714055}
I20251026 02:56:43.754405 1980 raft_consensus.cc:3060] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:43.754431 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 8688a0bf946841deb5d88c931538b129 (table default.Table_a6e8714ddccb4139adb6c8320e06500b [id=bb34b75c6fcb4ea6b201d05be4982a75]) successfully deleted
I20251026 02:56:43.758975 2375 tablet_bootstrap.cc:492] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:43.760485 2375 ts_tablet_manager.cc:1403] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.003s
I20251026 02:56:43.760905 3877 consensus_queue.cc:1048] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:43.762713 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa10cd990a944eab8d9eb66aaf20080c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:43.763305 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eafb10b378e34ed3b4a33e56c6600759 has no consensus state
W20251026 02:56:43.766527 2194 leader_election.cc:343] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:43.766748 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24449cb9121642009dec41220e27b193 consensus state has no leader
I20251026 02:56:43.769726 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 753) took 5936 ms. Trace:
I20251026 02:56:43.769861 2377 rpcz_store.cc:276] 1026 02:56:37.832713 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.918627 (+ 85914us) service_pool.cc:225] Handling call
1026 02:56:43.754268 (+5835641us) ts_tablet_manager.cc:1151] Deleting tablet 0ddeba351c264f67b75b66c55c9734b5
1026 02:56:43.768699 (+ 14431us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.769279 (+ 580us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5829591}
I20251026 02:56:43.770229 2378 tablet_bootstrap.cc:654] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:43.770809 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0ddeba351c264f67b75b66c55c9734b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ddeba351c264f67b75b66c55c9734b5 already in progress: creating tablet
I20251026 02:56:43.775302 2376 ts_tablet_manager.cc:1929] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:43.775473 2376 log.cc:1199] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/a378e913112c443db869016e3b98fd0e
I20251026 02:56:43.775868 2376 ts_tablet_manager.cc:1950] T a378e913112c443db869016e3b98fd0e P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:43.776197 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 773) took 4190 ms. Trace:
I20251026 02:56:43.776335 2376 rpcz_store.cc:276] 1026 02:56:39.585545 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.585609 (+ 64us) service_pool.cc:225] Handling call
1026 02:56:43.576731 (+3991122us) ts_tablet_manager.cc:1151] Deleting tablet a378e913112c443db869016e3b98fd0e
1026 02:56:43.576745 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.582027 (+ 5282us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.775195 (+193168us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.776182 (+ 987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3990878}
I20251026 02:56:43.776714 2376 tablet_replica.cc:333] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:43.776918 2376 raft_consensus.cc:2243] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:43.777107 2376 raft_consensus.cc:2272] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.777746 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet a378e913112c443db869016e3b98fd0e (table default.Table_f2809b812fe14b2d9e44b22bfa68c1f7 [id=f83014c25cfa449d976ead67854d2ca7]) successfully deleted
I20251026 02:56:43.778040 2375 raft_consensus.cc:359] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.779551 2375 raft_consensus.cc:385] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.780496 2375 raft_consensus.cc:740] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.783766 3877 consensus_queue.cc:1048] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:43.785463 3841 raft_consensus.cc:493] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.785584 3841 raft_consensus.cc:515] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.786129 3841 leader_election.cc:290] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.787006 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d43b6f22dd4f4f9c9657f21514550312" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:43.787585 2053 leader_election.cc:343] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.789660 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d43b6f22dd4f4f9c9657f21514550312" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.790050 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 762) took 5856 ms. Trace:
I20251026 02:56:43.790148 2377 rpcz_store.cc:276] 1026 02:56:37.934036 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.934081 (+ 45us) service_pool.cc:225] Handling call
1026 02:56:43.770140 (+5836059us) ts_tablet_manager.cc:1151] Deleting tablet 3aa7a90a29ec430286786bf8aa9d4c02
1026 02:56:43.789552 (+ 19412us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.790035 (+ 483us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5835850}
W20251026 02:56:43.790624 2053 leader_election.cc:343] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.790731 2053 leader_election.cc:304] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.791134 3841 raft_consensus.cc:2749] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.793767 3846 raft_consensus.cc:493] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.793910 3846 raft_consensus.cc:515] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.794438 2377 tablet_replica.cc:333] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:43.794518 3846 leader_election.cc:290] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.794765 2377 raft_consensus.cc:2243] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:43.795310 2377 raft_consensus.cc:2272] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.796344 2377 ts_tablet_manager.cc:1916] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:43.796371 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a00bc04b3654f90af342986751344e6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:43.796934 2053 leader_election.cc:343] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.798389 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a00bc04b3654f90af342986751344e6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:43.799207 2053 leader_election.cc:343] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.799350 2053 leader_election.cc:304] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.799712 3841 raft_consensus.cc:2749] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:43.802609 1648 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 405680us
I20251026 02:56:43.803231 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet f719c720cc474181a63786a3e248ab2c (table default.Table_e586d81375544236ac7d95297fca01a8 [id=67731996dfc04572b0dfebf0edb71942]) successfully deleted
W20251026 02:56:43.803519 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 58d60bf2a9e546e6a9474a5af6d5273a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58d60bf2a9e546e6a9474a5af6d5273a already in progress: creating tablet
W20251026 02:56:43.803797 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 63a63b796191465aab9a63d3ffb0554d because the tablet was not found. No further retry: Not found: Tablet not found: 63a63b796191465aab9a63d3ffb0554d
W20251026 02:56:43.804029 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 244c03a9293242a48077bdaf90a47865 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 244c03a9293242a48077bdaf90a47865 already in progress: creating tablet
I20251026 02:56:43.783185 2375 consensus_queue.cc:260] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.811201 2381 tablet_bootstrap.cc:492] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:43.811483 2381 ts_tablet_manager.cc:1403] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.089s user 0.005s sys 0.001s
I20251026 02:56:43.811761 2375 ts_tablet_manager.cc:1434] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.050s user 0.002s sys 0.001s
I20251026 02:56:43.812925 2381 raft_consensus.cc:359] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.813247 2381 raft_consensus.cc:385] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.813383 2381 raft_consensus.cc:740] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.814985 2375 tablet_bootstrap.cc:492] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:43.817143 3496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Bootstrap replaying log segment 1/1 (72.9K/75.5K this segment, stats: ops{read=408 overwritten=1 applied=348 ignored=0} inserts{seen=331 ignored=0} mutations{seen=395 ignored=0} orphaned_commits=0)
W20251026 02:56:43.820386 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ded06ba1f80c4433ae78687e5990edce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ded06ba1f80c4433ae78687e5990edce already in progress: creating tablet
I20251026 02:56:43.820706 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 (table default.Table_2dfe8bcc7f1e4933a4174bdab19cffd5 [id=65d576adb71949cc97666624935adbf3]) successfully deleted
W20251026 02:56:43.820971 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0fe0e06a86e241bc91a1db4219fe25bc because the tablet was not found. No further retry: Not found: Tablet not found: 0fe0e06a86e241bc91a1db4219fe25bc
I20251026 02:56:43.821190 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7801f073d5d94076af9299492f86fad2 (table default.Table_03a9f618938a45aab881c91a801cc32c [id=12b41b47b5d243188eb73bd83a83bbdc]) successfully deleted
W20251026 02:56:43.821448 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 1a36b8a4beca483bb853d27961f5281a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a36b8a4beca483bb853d27961f5281a already in progress: creating tablet
W20251026 02:56:43.821692 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet b68388b2fd014dd1a5f6572796cf53a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b68388b2fd014dd1a5f6572796cf53a6 already in progress: creating tablet
W20251026 02:56:43.821938 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 81775055d98349f696cdab6171ffd58b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81775055d98349f696cdab6171ffd58b already in progress: creating tablet
W20251026 02:56:43.822152 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7e1a45412db1449a99252112d8110a17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1a45412db1449a99252112d8110a17 already in progress: creating tablet
W20251026 02:56:43.822458 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet e32a1f8dcad4430eb35d388bfad7d110 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e32a1f8dcad4430eb35d388bfad7d110 already in progress: creating tablet
W20251026 02:56:43.822698 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet b8cad7adae7e43469ceadbedeee53564 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8cad7adae7e43469ceadbedeee53564 already in progress: creating tablet
W20251026 02:56:43.822935 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ba08469339184b11b9ced9e6bdd63d42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba08469339184b11b9ced9e6bdd63d42 already in progress: creating tablet
I20251026 02:56:43.823513 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet bcd5621aa3754733b22508be89f080db with delete_type TABLET_DATA_DELETED (replaced by tablet c34406d36c6e48dc973584bd653a9e42) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.826611 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fdadd053cc34aeca0d491eb3200207f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.827436 2378 tablet_bootstrap.cc:492] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:43.827656 2378 ts_tablet_manager.cc:1403] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.003s
I20251026 02:56:43.828444 2378 raft_consensus.cc:359] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.828651 2378 raft_consensus.cc:385] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.828716 2378 raft_consensus.cc:740] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.828950 2378 consensus_queue.cc:260] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.830318 3866 tablet.cc:1722] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:43.830760 3372 raft_consensus.cc:493] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.830940 3372 raft_consensus.cc:515] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.831671 3372 leader_election.cc:290] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:43.832281 1916 leader_election.cc:343] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 3dbe8002df6c4b64b8ec8260ca6b9cf5
I20251026 02:56:43.837076 1659 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet de8799a5af2b42cc92ccc0195cff98e7 (table default.Table_13667e60be9c46b192cebf20df523e99 [id=7885bd43f8a245b0a3010a63dbd535f5]). Expected version 3 got 0
I20251026 02:56:43.837291 2378 ts_tablet_manager.cc:1434] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251026 02:56:43.841179 2378 tablet_bootstrap.cc:492] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:43.844659 1661 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet de8799a5af2b42cc92ccc0195cff98e7 (table default.Table_13667e60be9c46b192cebf20df523e99 [id=7885bd43f8a245b0a3010a63dbd535f5]). Expected version 3 got 0
I20251026 02:56:43.826694 2381 consensus_queue.cc:260] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.839614 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dbe8002df6c4b64b8ec8260ca6b9cf5" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.853971 2372 ts_tablet_manager.cc:1929] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:43.855015 2381 ts_tablet_manager.cc:1434] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20251026 02:56:43.855280 2372 log.cc:1199] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/6108219254bd448c852096c0c967384b
I20251026 02:56:43.855624 2378 tablet_bootstrap.cc:654] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.855629 2381 tablet_bootstrap.cc:492] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:43.856168 2375 tablet_bootstrap.cc:654] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:43.791590 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3aa7a90a29ec430286786bf8aa9d4c02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3aa7a90a29ec430286786bf8aa9d4c02 already in progress: creating tablet
W20251026 02:56:43.857029 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 65416.1us
I20251026 02:56:43.858018 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_88c37cad8c6841d69e4ed80515b92b5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:43.858381 2372 ts_tablet_manager.cc:1950] T 6108219254bd448c852096c0c967384b P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:43.858639 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88c37cad8c6841d69e4ed80515b92b5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.858731 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3b00d3dd0fe54a4b867f4dcf830d98a9" } new_table_name: "default.Table_6c6a682362754ce1be7ef856686520e0" modify_external_catalogs: true
I20251026 02:56:43.859126 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 882cd1680aa140f6a63a961cd4aec25a (table default.Table_52417f25dbdb4676a5aaaca9bf46962c [id=cc7a8c37a3b342d988872e3baa16a26d]). Expected version 1 got 0
I20251026 02:56:43.860235 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6bf9cedc1f4c46479c8d1b407d1fdc5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:43.860347 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 882cd1680aa140f6a63a961cd4aec25a (table default.Table_52417f25dbdb4676a5aaaca9bf46962c [id=cc7a8c37a3b342d988872e3baa16a26d]). Expected version 1 got 0
W20251026 02:56:43.860769 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bf9cedc1f4c46479c8d1b407d1fdc5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.861017 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 743) took 6486 ms. Trace:
I20251026 02:56:43.862586 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_09b68d3ebab94c55b3c90567af20db3d" } modify_external_catalogs: true
I20251026 02:56:43.862818 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_09b68d3ebab94c55b3c90567af20db3d" } modify_external_catalogs: true
I20251026 02:56:43.862146 2372 rpcz_store.cc:276] 1026 02:56:37.374521 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.374578 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:43.744568 (+6369990us) ts_tablet_manager.cc:1151] Deleting tablet 6108219254bd448c852096c0c967384b
1026 02:56:43.744576 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.773743 (+ 29167us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.844182 (+ 70439us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.860773 (+ 16591us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6369707}
I20251026 02:56:43.863692 1915 leader_election.cc:304] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:43.864158 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 6108219254bd448c852096c0c967384b (table default.Table_5af1941b0082444c8bd348a080af20ee [id=6342ae7b8acf4137a16ebf978ce7e15b]) successfully deleted
I20251026 02:56:43.864393 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet b1b6613742a24c21a1db6ebe858a846f from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:43.864547 3372 raft_consensus.cc:2749] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:43.865124 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fde9bb8a8be244ed9462ed7367173e96" } modify_external_catalogs: true
I20251026 02:56:43.865324 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fde9bb8a8be244ed9462ed7367173e96" } modify_external_catalogs: true
I20251026 02:56:43.865731 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 746) took 6436 ms. Trace:
I20251026 02:56:43.866190 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3fb4e68115eb4b9f98a57da13662a443"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.866711 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fb4e68115eb4b9f98a57da13662a443 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.866058 2372 rpcz_store.cc:276] 1026 02:56:37.428730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.430000 (+ 1270us) service_pool.cc:225] Handling call
1026 02:56:43.864912 (+6434912us) ts_tablet_manager.cc:1151] Deleting tablet 7f5cd37c87f34a40bf05829af6de0473
1026 02:56:43.864924 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.865544 (+ 620us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6426844}
W20251026 02:56:43.867525 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7f5cd37c87f34a40bf05829af6de0473 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f5cd37c87f34a40bf05829af6de0473 already in progress: creating tablet
I20251026 02:56:43.869612 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 748) took 6366 ms. Trace:
W20251026 02:56:43.870604 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e7c609238394fb7ab1597556894774a consensus state has no leader
I20251026 02:56:43.870152 2372 rpcz_store.cc:276] 1026 02:56:37.503237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.503347 (+ 110us) service_pool.cc:225] Handling call
1026 02:56:43.868965 (+6365618us) ts_tablet_manager.cc:1151] Deleting tablet 8aa0575af07e433c8d96acab31edb4af
1026 02:56:43.868977 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.869592 (+ 615us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6365257}
I20251026 02:56:43.870956 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9e61f50aaa744075aa0488e84f1166b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.871433 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 8aa0575af07e433c8d96acab31edb4af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aa0575af07e433c8d96acab31edb4af already in progress: creating tablet
W20251026 02:56:43.871475 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e61f50aaa744075aa0488e84f1166b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.872937 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 755) took 6275 ms. Trace:
I20251026 02:56:43.873546 2372 rpcz_store.cc:276] 1026 02:56:37.597084 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.616090 (+ 19006us) service_pool.cc:225] Handling call
1026 02:56:43.872122 (+6256032us) ts_tablet_manager.cc:1151] Deleting tablet 6979455632e54d2081065954771f34d3
1026 02:56:43.872137 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.872915 (+ 778us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6255507}
W20251026 02:56:43.874785 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 6979455632e54d2081065954771f34d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6979455632e54d2081065954771f34d3 already in progress: creating tablet
I20251026 02:56:43.876096 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 756) took 6233 ms. Trace:
I20251026 02:56:43.876500 2372 rpcz_store.cc:276] 1026 02:56:37.642571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.696282 (+ 53711us) service_pool.cc:225] Handling call
1026 02:56:43.875389 (+6179107us) ts_tablet_manager.cc:1151] Deleting tablet d7313a6396c541d7ac6d8ca2e219af90
1026 02:56:43.875582 (+ 193us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.875931 (+ 349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6176615}
I20251026 02:56:43.878470 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_03342f2363b44f6599f1d4cc6fef2be8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:43.879338 2372 tablet_replica.cc:333] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:43.878018 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet d7313a6396c541d7ac6d8ca2e219af90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7313a6396c541d7ac6d8ca2e219af90 already in progress: creating tablet
W20251026 02:56:43.882344 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03342f2363b44f6599f1d4cc6fef2be8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.881814 2372 raft_consensus.cc:2243] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:43.884704 2375 tablet_bootstrap.cc:492] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:43.884935 2375 ts_tablet_manager.cc:1403] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.001s
I20251026 02:56:43.885443 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet f719c720cc474181a63786a3e248ab2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:43.886181 2375 raft_consensus.cc:359] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.886503 2375 raft_consensus.cc:385] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.886621 2375 raft_consensus.cc:740] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.886948 2375 consensus_queue.cc:260] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.888298 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_81d15a2f4d424910a51421472c159f0d" } modify_external_catalogs: true
I20251026 02:56:43.888484 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_81d15a2f4d424910a51421472c159f0d" } modify_external_catalogs: true
I20251026 02:56:43.890337 2027 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:43.891450 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b061587f8f69417d870501f300207da4" } new_table_name: "default.Table_3ea9b77cd2cc43b99bdf4ab495a678d9" modify_external_catalogs: true
I20251026 02:56:43.894183 2375 ts_tablet_manager.cc:1434] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20251026 02:56:43.894599 2375 tablet_bootstrap.cc:492] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:43.898517 2372 raft_consensus.cc:2272] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:43.901378 2377 ts_tablet_manager.cc:1929] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:43.902384 2378 tablet_bootstrap.cc:492] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:43.902565 2378 ts_tablet_manager.cc:1403] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20251026 02:56:43.903338 2378 raft_consensus.cc:359] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.903544 2378 raft_consensus.cc:385] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.903627 2378 raft_consensus.cc:740] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.903875 2378 consensus_queue.cc:260] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.908187 2372 ts_tablet_manager.cc:1916] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:43.909862 2378 ts_tablet_manager.cc:1434] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251026 02:56:43.910182 2378 tablet_bootstrap.cc:492] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:43.910919 2375 tablet_bootstrap.cc:654] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.911737 1661 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_13667e60be9c46b192cebf20df523e99 [id=7885bd43f8a245b0a3010a63dbd535f5] alter complete (version 3)
I20251026 02:56:43.912945 1657 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 882cd1680aa140f6a63a961cd4aec25a (table default.Table_52417f25dbdb4676a5aaaca9bf46962c [id=cc7a8c37a3b342d988872e3baa16a26d]). Expected version 1 got 0
W20251026 02:56:43.913300 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa1253c60ffb4b37849c9318307b94f1 consensus state has no leader
I20251026 02:56:43.911828 3841 raft_consensus.cc:493] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:43.915230 3841 raft_consensus.cc:515] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.915913 3841 leader_election.cc:290] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:43.912521 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet d43b6f22dd4f4f9c9657f21514550312 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.912943 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b5e61afe90e4bc4a2897de21e0b3919 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b562b7b51f94b268c667650df199fbd) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.913159 3866 tablet.cc:1717] T de8799a5af2b42cc92ccc0195cff98e7 P 0b6b0a6c4e144982b67980118cddba48: Skipping requested alter to schema version 3, tablet already version 3
I20251026 02:56:43.913441 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet f2d97084b5914350afbfda5c2a34bf9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.917747 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d97084b5914350afbfda5c2a34bf9a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:43.918515 2053 leader_election.cc:343] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.918931 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_03f29b21370a4ee8b75f3f37f6666e75" } new_table_name: "default.Table_0d155486332941679014bf465dbbb2de" modify_external_catalogs: true
I20251026 02:56:43.919417 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fa73267dd5574a2cbc440226ab6cbb40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:43.922266 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet e89a98fcfdbb4011876b0ebc2af91f67 with delete_type TABLET_DATA_DELETED (replaced by tablet d0f1d6967931482e9de40b0844da6315) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.927629 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_94dc93bcf0ca4d58bd2ab854fba805a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.929797 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa73267dd5574a2cbc440226ab6cbb40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.931113 2378 tablet_bootstrap.cc:654] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:43.930946 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94dc93bcf0ca4d58bd2ab854fba805a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.936905 2377 log.cc:1199] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/1370b9ad917344c093e5da3d0e9b1cf0
I20251026 02:56:43.910822 3904 tablet.cc:1722] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:43.919607 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d97084b5914350afbfda5c2a34bf9a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:43.933462 2381 tablet_bootstrap.cc:654] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.939627 2377 ts_tablet_manager.cc:1950] T 1370b9ad917344c093e5da3d0e9b1cf0 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:43.941874 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 759) took 6008 ms. Trace:
I20251026 02:56:43.943531 2375 tablet_bootstrap.cc:492] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:43.943763 2375 ts_tablet_manager.cc:1403] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.002s
I20251026 02:56:43.942468 2377 rpcz_store.cc:276] 1026 02:56:37.933660 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.935981 (+ 2321us) service_pool.cc:225] Handling call
1026 02:56:43.790447 (+5854466us) ts_tablet_manager.cc:1151] Deleting tablet 1370b9ad917344c093e5da3d0e9b1cf0
1026 02:56:43.794342 (+ 3895us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.865500 (+ 71158us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.901264 (+ 35764us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.941685 (+ 40421us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5845750}
I20251026 02:56:43.944767 2375 raft_consensus.cc:359] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.944965 2375 raft_consensus.cc:385] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.945111 2375 raft_consensus.cc:740] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.945446 2375 consensus_queue.cc:260] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.946147 2375 ts_tablet_manager.cc:1434] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:43.955417 2375 tablet_bootstrap.cc:492] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:43.958210 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet dfd3e23d83a84648bd11cd258377aecb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:43.961944 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_137bc0aed1ba4f0ebe2575519bea40a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:43.963912 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 1370b9ad917344c093e5da3d0e9b1cf0 (table default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]) successfully deleted
W20251026 02:56:43.966477 2053 leader_election.cc:343] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:43.966629 2053 leader_election.cc:304] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:43.966878 3841 raft_consensus.cc:2749] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:43.967057 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_137bc0aed1ba4f0ebe2575519bea40a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.968130 2375 tablet_bootstrap.cc:654] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:43.970296 1662 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 882cd1680aa140f6a63a961cd4aec25a (table default.Table_52417f25dbdb4676a5aaaca9bf46962c [id=cc7a8c37a3b342d988872e3baa16a26d]). Expected version 1 got 0
I20251026 02:56:43.971115 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_860f88c3c76242d28c07e73bcbb7d077"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:43.971748 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_860f88c3c76242d28c07e73bcbb7d077 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:43.971823 2372 ts_tablet_manager.cc:1929] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:43.972642 2372 log.cc:1199] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/8688a0bf946841deb5d88c931538b129
I20251026 02:56:43.977483 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 760) took 6043 ms. Trace:
I20251026 02:56:43.978354 2377 rpcz_store.cc:276] 1026 02:56:37.933791 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.944517 (+ 10726us) service_pool.cc:225] Handling call
1026 02:56:43.945762 (+6001245us) ts_tablet_manager.cc:1151] Deleting tablet b1b6613742a24c21a1db6ebe858a846f
1026 02:56:43.945778 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.977116 (+ 31338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6000430}
I20251026 02:56:43.981778 2376 ts_tablet_manager.cc:1916] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:43.982803 2377 tablet_replica.cc:333] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:43.983287 2378 tablet_bootstrap.cc:492] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:43.985001 2378 ts_tablet_manager.cc:1403] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.075s user 0.004s sys 0.003s
I20251026 02:56:43.986281 3904 tablet.cc:1717] T de8799a5af2b42cc92ccc0195cff98e7 P 696476265495492790739ec126a08539: Skipping requested alter to schema version 3, tablet already version 3
I20251026 02:56:43.989140 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet 63f1abc4477f4198ad2cc8481d69df73 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:43.989861 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet b1b6613742a24c21a1db6ebe858a846f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1b6613742a24c21a1db6ebe858a846f already in progress: creating tablet
I20251026 02:56:43.990948 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a905e23c621542fa802270f96a8dfd8f" } new_table_name: "default.Table_9cbff204e967439e869756f29385106a" modify_external_catalogs: true
I20251026 02:56:43.994472 2378 raft_consensus.cc:359] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:43.994721 2378 raft_consensus.cc:385] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:43.994802 2378 raft_consensus.cc:740] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:43.995087 2378 consensus_queue.cc:260] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:44.001343 3496 log.cc:927] Time spent T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Append to log took a long time: real 0.118s user 0.003s sys 0.001s
I20251026 02:56:43.985216 2377 raft_consensus.cc:2243] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:44.010748 3372 raft_consensus.cc:493] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.010903 3372 raft_consensus.cc:515] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.011030 2377 raft_consensus.cc:2272] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.011516 3372 leader_election.cc:290] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:44.011616 2377 ts_tablet_manager.cc:1916] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.012173 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8799a5af2b42cc92ccc0195cff98e7" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:44.013325 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8799a5af2b42cc92ccc0195cff98e7" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:44.013793 1915 leader_election.cc:304] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:44.014074 3372 raft_consensus.cc:2749] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:44.020982 1975 raft_consensus.cc:3060] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.023976 1976 raft_consensus.cc:3060] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.025326 2376 ts_tablet_manager.cc:1929] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:44.028337 2376 log.cc:1199] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3327d26a2c2c4d0b86d98151f5940bd7
W20251026 02:56:44.028344 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e91466733c4142a7b2879334c0b67aaa consensus state has no leader
I20251026 02:56:44.028733 2376 ts_tablet_manager.cc:1950] T 3327d26a2c2c4d0b86d98151f5940bd7 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:44.029080 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 774) took 4442 ms. Trace:
I20251026 02:56:44.029199 2376 rpcz_store.cc:276] 1026 02:56:39.586510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.586565 (+ 55us) service_pool.cc:225] Handling call
1026 02:56:43.776666 (+4190101us) ts_tablet_manager.cc:1151] Deleting tablet 3327d26a2c2c4d0b86d98151f5940bd7
1026 02:56:43.776675 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.984759 (+208084us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.025218 (+ 40459us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.029064 (+ 3846us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4189898}
I20251026 02:56:44.029628 2376 tablet_replica.cc:333] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:44.029839 2372 ts_tablet_manager.cc:1950] T 8688a0bf946841deb5d88c931538b129 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:44.030895 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 764) took 6206 ms. Trace:
I20251026 02:56:44.031157 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_21830ac7f7ef4cfdb68626b890c460b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:44.036792 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 882fe81da21344ed99ea735fa79f73c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:44.042157 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b22d8322842a4ff6ad61fc3a8ad64f62" } new_table_name: "default.Table_c4061134bbfd4244a23d6c37781dcb86" modify_external_catalogs: true
I20251026 02:56:44.041767 2378 ts_tablet_manager.cc:1434] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.048s user 0.001s sys 0.001s
I20251026 02:56:44.048871 2378 tablet_bootstrap.cc:492] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:44.050300 1982 raft_consensus.cc:3060] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.053086 2376 raft_consensus.cc:2243] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:44.053536 2376 raft_consensus.cc:2272] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.054919 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3327d26a2c2c4d0b86d98151f5940bd7 (table default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 [id=7de20c0869584b56b936144d62202444]) successfully deleted
I20251026 02:56:44.057940 2376 ts_tablet_manager.cc:1916] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:44.060550 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21830ac7f7ef4cfdb68626b890c460b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:44.064337 2378 tablet_bootstrap.cc:654] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.068892 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet 8977f824ff374d05b4ed9d88c67329f9 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:44.072175 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_279edde3f26a49bcbf0c6c501709fadc" } new_table_name: "default.Table_b9ae3b0edd284b839df1afb9b661c6fc" modify_external_catalogs: true
I20251026 02:56:44.057006 3841 raft_consensus.cc:493] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.074304 3841 raft_consensus.cc:515] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.074994 3841 leader_election.cc:290] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:44.075743 2372 rpcz_store.cc:276] 1026 02:56:37.824531 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.824581 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:43.878778 (+6054197us) ts_tablet_manager.cc:1151] Deleting tablet 8688a0bf946841deb5d88c931538b129
1026 02:56:43.878790 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:43.947884 (+ 69094us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:43.970939 (+ 23055us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.030838 (+ 59899us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6053846}
I20251026 02:56:44.076308 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 758) took 6341 ms. Trace:
I20251026 02:56:44.076475 2372 rpcz_store.cc:276] 1026 02:56:37.734912 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.736064 (+ 1152us) service_pool.cc:225] Handling call
1026 02:56:44.076127 (+6340063us) ts_tablet_manager.cc:1151] Deleting tablet 0ddeba351c264f67b75b66c55c9734b5
1026 02:56:44.076139 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.076268 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6245401}
I20251026 02:56:44.076715 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfd3e23d83a84648bd11cd258377aecb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:44.076917 2372 tablet_replica.cc:333] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:44.077320 2053 leader_election.cc:343] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.077710 2372 raft_consensus.cc:2243] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:44.078227 2372 raft_consensus.cc:2272] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.079448 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfd3e23d83a84648bd11cd258377aecb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:44.079892 2372 ts_tablet_manager.cc:1916] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:44.080070 2053 leader_election.cc:343] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.080235 2053 leader_election.cc:304] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:44.080626 3841 raft_consensus.cc:2749] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:44.080896 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 8688a0bf946841deb5d88c931538b129 (table default.Table_a6e8714ddccb4139adb6c8320e06500b [id=bb34b75c6fcb4ea6b201d05be4982a75]) successfully deleted
W20251026 02:56:44.081179 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0ddeba351c264f67b75b66c55c9734b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ddeba351c264f67b75b66c55c9734b5 already in progress: creating tablet
I20251026 02:56:44.082142 2376 ts_tablet_manager.cc:1929] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.084052 2376 log.cc:1199] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/ce922040298d4952af13652eeda7a234
W20251026 02:56:44.085323 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b4bc725e76c41bcab92e87c9b85f95d consensus state has no leader
I20251026 02:56:44.085513 2376 ts_tablet_manager.cc:1950] T ce922040298d4952af13652eeda7a234 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:44.090246 3904 tablet.cc:1722] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.092403 2381 tablet_bootstrap.cc:492] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:44.092612 2381 ts_tablet_manager.cc:1403] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.237s user 0.005s sys 0.003s
I20251026 02:56:44.093070 3913 tablet.cc:1722] T 8adf5c6970be46ed8434fd28404b2166 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.093478 2381 raft_consensus.cc:359] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.093711 2381 raft_consensus.cc:385] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:44.093861 2381 raft_consensus.cc:740] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.094173 2381 consensus_queue.cc:260] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.095295 2381 ts_tablet_manager.cc:1434] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:44.100780 3928 tablet.cc:1722] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:44.104985 2381 tablet_bootstrap.cc:492] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:44.109746 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 23062e7292f44ad5ad9dfc22a401741c (table default.Table_11cf708e0e8243c887777e42acc25490 [id=2e17bae10e0b48a492f5ee0bed7d00d0]). Expected version 1 got 0
I20251026 02:56:44.110133 2378 tablet_bootstrap.cc:492] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:44.110952 2378 ts_tablet_manager.cc:1403] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.002s
I20251026 02:56:44.120064 2378 raft_consensus.cc:359] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.124728 2381 tablet_bootstrap.cc:654] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.124905 2375 tablet_bootstrap.cc:492] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:44.128518 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7e21560192e242298794bb889dfcc622"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:44.129000 2378 raft_consensus.cc:385] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251026 02:56:44.129194 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e21560192e242298794bb889dfcc622 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:44.126315 2375 ts_tablet_manager.cc:1403] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.171s user 0.009s sys 0.003s
I20251026 02:56:44.135040 2375 raft_consensus.cc:359] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.141901 2378 raft_consensus.cc:740] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.122077 3929 tablet.cc:1722] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.118494 3925 tablet.cc:1722] T 8adf5c6970be46ed8434fd28404b2166 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.145084 3904 tablet.cc:1722] T f7da859119bd42ea9fa1290b3d4bf88c P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.147276 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3447a540091c4190a201673b0fc1fa4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:44.147948 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3447a540091c4190a201673b0fc1fa4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:44.148331 3890 raft_consensus.cc:493] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.148465 3890 raft_consensus.cc:515] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.149077 3890 leader_election.cc:290] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:44.149175 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 775) took 4548 ms. Trace:
I20251026 02:56:44.149328 2376 rpcz_store.cc:276] 1026 02:56:39.601123 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.601192 (+ 69us) service_pool.cc:225] Handling call
1026 02:56:44.029583 (+4428391us) ts_tablet_manager.cc:1151] Deleting tablet ce922040298d4952af13652eeda7a234
1026 02:56:44.029592 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.062537 (+ 32945us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.082027 (+ 19490us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.149156 (+ 67129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4428107}
I20251026 02:56:44.150039 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet ce922040298d4952af13652eeda7a234 (table default.Table_0ea3c4e29f264b34b9f6030b92262af6 [id=d33bf698f19d407da868aa876f64de5e]) successfully deleted
I20251026 02:56:44.150796 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0afb6f04f5ac463780fe8f522bf2fe23
I20251026 02:56:44.151672 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_57ce21096a194256b1f340c52bc836a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:44.152134 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 0afb6f04f5ac463780fe8f522bf2fe23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:44.152231 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57ce21096a194256b1f340c52bc836a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:44.152597 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 0afb6f04f5ac463780fe8f522bf2fe23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:44.153012 2376 tablet_replica.cc:333] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:44.153304 2376 raft_consensus.cc:2243] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:44.153532 2376 pending_rounds.cc:70] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8: Trying to abort 1 pending ops.
I20251026 02:56:44.153640 2376 pending_rounds.cc:77] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7214888557558775808 op_type: NO_OP noop_request { }
I20251026 02:56:44.153888 2376 raft_consensus.cc:2889] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:44.154462 2376 raft_consensus.cc:2272] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.162101 3935 tablet.cc:1722] T f7da859119bd42ea9fa1290b3d4bf88c P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.163746 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 146d6f13b9ef47f29711a102791bab5a
I20251026 02:56:44.164760 3890 raft_consensus.cc:493] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:44.164949 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0c60d8227eed417595ef68d44c68ebcf
I20251026 02:56:44.164916 3890 raft_consensus.cc:515] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:44.165517 3890 leader_election.cc:290] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:44.166836 2195 leader_election.cc:343] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ae95654d129441e08fcb3f3b355ffc22
I20251026 02:56:44.168403 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 146d6f13b9ef47f29711a102791bab5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:44.168494 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1
I20251026 02:56:44.168624 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c60d8227eed417595ef68d44c68ebcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:44.168998 1656 catalog_manager.cc:6719] ReplaceTablet: tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 deleted and replaced by tablet 69885e58f9e04979b6157c61e4c0f261
I20251026 02:56:44.170361 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 63f1abc4477f4198ad2cc8481d69df73
I20251026 02:56:44.170893 1661 catalog_manager.cc:6719] ReplaceTablet: tablet 63f1abc4477f4198ad2cc8481d69df73 deleted and replaced by tablet 5cab1d6e8d8048f791ec23b78d276848
I20251026 02:56:44.171236 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5068032512e84748bf5f2399a08bfa05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:44.171556 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 69885e58f9e04979b6157c61e4c0f261) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:44.171734 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 63f1abc4477f4198ad2cc8481d69df73 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cab1d6e8d8048f791ec23b78d276848) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:44.164034 2376 ts_tablet_manager.cc:1916] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.166003 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23062e7292f44ad5ad9dfc22a401741c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:44.171883 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5068032512e84748bf5f2399a08bfa05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:44.165988 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae95654d129441e08fcb3f3b355ffc22" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:44.167366 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 146d6f13b9ef47f29711a102791bab5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:44.167883 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c60d8227eed417595ef68d44c68ebcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:44.170229 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 69885e58f9e04979b6157c61e4c0f261) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:44.170981 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 63f1abc4477f4198ad2cc8481d69df73 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cab1d6e8d8048f791ec23b78d276848) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:44.172389 2194 leader_election.cc:343] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.174147 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 5eda85f2d53a43d68907c467c7da9bd6
I20251026 02:56:44.174365 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f44743d6a730429da0c9d4900b298c9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:44.174791 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 5eda85f2d53a43d68907c467c7da9bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:44.174904 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f44743d6a730429da0c9d4900b298c9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:44.174847 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d4fb1ce9057f4cde849a4a8050bf6a18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:44.175292 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4fb1ce9057f4cde849a4a8050bf6a18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:44.175618 2378 consensus_queue.cc:260] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.176096 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 5eda85f2d53a43d68907c467c7da9bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:44.176203 2378 ts_tablet_manager.cc:1434] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.057s user 0.000s sys 0.001s
I20251026 02:56:44.176488 2378 tablet_bootstrap.cc:492] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:44.176507 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a9e832f0f3764a788f4a90cb03b5b19b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:44.176937 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9e832f0f3764a788f4a90cb03b5b19b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:44.178697 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7a84416153d6424eb94a56932605f7ad" } new_table_name: "default.Table_e6e84b24da5b4c8eb22ab71b440d5a0f" modify_external_catalogs: true
I20251026 02:56:44.153260 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23062e7292f44ad5ad9dfc22a401741c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:44.154014 2099 tablet_service.cc:1552] Processing DeleteTablet for tablet 0afb6f04f5ac463780fe8f522bf2fe23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:44.182708 2114 raft_consensus.cc:2468] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:44.166453 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 146d6f13b9ef47f29711a102791bab5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:44.166600 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c60d8227eed417595ef68d44c68ebcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:44.169049 3866 tablet.cc:1722] T f7da859119bd42ea9fa1290b3d4bf88c P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.178435 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 69885e58f9e04979b6157c61e4c0f261) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:44.178613 2083 tablet_service.cc:1552] Processing DeleteTablet for tablet 63f1abc4477f4198ad2cc8481d69df73 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cab1d6e8d8048f791ec23b78d276848) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:44.178745 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 5eda85f2d53a43d68907c467c7da9bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:44.186715 2195 leader_election.cc:304] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:44.189811 3890 raft_consensus.cc:2804] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:44.189913 3890 raft_consensus.cc:493] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:44.189980 3890 raft_consensus.cc:3060] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.190729 3904 tablet.cc:1722] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.130745 2377 ts_tablet_manager.cc:1929] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.191354 2377 log.cc:1199] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3c3050e72d42417ab58a05f7e3aa56b3
I20251026 02:56:44.191788 2377 ts_tablet_manager.cc:1950] T 3c3050e72d42417ab58a05f7e3aa56b3 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:44.177116 1980 raft_consensus.cc:2468] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:44.180666 3372 raft_consensus.cc:493] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.192961 3372 raft_consensus.cc:515] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.193626 3372 leader_election.cc:290] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:44.194382 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882cd1680aa140f6a63a961cd4aec25a" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:44.194425 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882cd1680aa140f6a63a961cd4aec25a" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:44.194952 1915 leader_election.cc:304] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:44.192090 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 764) took 6055 ms. Trace:
I20251026 02:56:44.195369 3372 raft_consensus.cc:2749] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:44.195804 3890 raft_consensus.cc:515] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.196411 2194 leader_election.cc:304] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:44.196493 3890 leader_election.cc:290] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:44.196614 3890 raft_consensus.cc:2804] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 02:56:44.196794 3890 raft_consensus.cc:493] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:44.196910 3890 raft_consensus.cc:3060] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:44.197124 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23062e7292f44ad5ad9dfc22a401741c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:44.197700 2194 leader_election.cc:343] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.195259 2377 rpcz_store.cc:276] 1026 02:56:38.136864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.137378 (+ 514us) service_pool.cc:225] Handling call
1026 02:56:43.982029 (+5844651us) ts_tablet_manager.cc:1151] Deleting tablet 3c3050e72d42417ab58a05f7e3aa56b3
1026 02:56:43.982279 (+ 250us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.063144 (+ 80865us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.130622 (+ 67478us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.192075 (+ 61453us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5844461}
I20251026 02:56:44.200244 3890 raft_consensus.cc:515] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:44.200872 3866 tablet.cc:1722] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.201310 3890 leader_election.cc:290] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:44.201748 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae95654d129441e08fcb3f3b355ffc22" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:44.202739 1980 raft_consensus.cc:3060] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 2
W20251026 02:56:44.203948 2195 leader_election.cc:343] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ae95654d129441e08fcb3f3b355ffc22
I20251026 02:56:44.204269 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3c3050e72d42417ab58a05f7e3aa56b3 (table default.Table_1fba8934d98d4fdb8b2c9a0b5d01e810 [id=b9b139445a9040398621353ad052f3c2]) successfully deleted
I20251026 02:56:44.204622 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23062e7292f44ad5ad9dfc22a401741c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:44.204829 2110 raft_consensus.cc:3060] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.207804 2110 raft_consensus.cc:2468] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:44.210568 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (table default.Table_30611e161c274e8b99bd1251e5e527aa [id=b5d91d3e8a2643c09fee85976972391a]). Expected version 1 got 0
I20251026 02:56:44.210959 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet ff242ef9541a42d793c4484190956da6 (table default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11]). Expected version 1 got 0
I20251026 02:56:44.215322 1873 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 146d6f13b9ef47f29711a102791bab5a
I20251026 02:56:44.215490 2378 tablet_bootstrap.cc:654] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.215700 2377 tablet_replica.cc:333] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:44.216390 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 146d6f13b9ef47f29711a102791bab5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:44.217190 2195 leader_election.cc:304] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:44.217425 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 146d6f13b9ef47f29711a102791bab5a (DEFAULT_TABLE table=default.Table_fde9bb8a8be244ed9462ed7367173e96 [id=4f930e9fc544463fad7425810a0f6337]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.217785 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146d6f13b9ef47f29711a102791bab5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.220346 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 8e2b23acac89427caef937831c4d2737 (DEFAULT_TABLE table=default.Table_d8455ca7030842aa80ba3d8da6c5a632 [id=5e643d181cc84f3eb8acf5449332100a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.220721 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2b23acac89427caef937831c4d2737. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.222869 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 92dd966c6bcc47d187cfdc9fcad7848a (DEFAULT_TABLE table=default.Table_b7298917793f498a8273bb74124d7807 [id=27394e7f480042199cd9397185b970d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.222987 3893 raft_consensus.cc:2804] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:44.223115 3893 raft_consensus.cc:697] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:44.223227 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92dd966c6bcc47d187cfdc9fcad7848a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.223362 3893 consensus_queue.cc:237] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.224530 2094 tablet_service.cc:1505] Processing CreateTablet for tablet f86766e836be46c3978917beb2797964 (DEFAULT_TABLE table=default.Table_7941853c36584b5591a75bf490bdb543 [id=f7002d452201483e829922a1b22277d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.224850 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f86766e836be46c3978917beb2797964. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.231145 2377 raft_consensus.cc:2243] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:44.223085 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 146d6f13b9ef47f29711a102791bab5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:44.225507 1954 tablet_service.cc:1505] Processing CreateTablet for tablet f86766e836be46c3978917beb2797964 (DEFAULT_TABLE table=default.Table_7941853c36584b5591a75bf490bdb543 [id=f7002d452201483e829922a1b22277d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.231948 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f86766e836be46c3978917beb2797964. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.236359 3841 raft_consensus.cc:493] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.236544 3841 raft_consensus.cc:515] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.226198 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 146d6f13b9ef47f29711a102791bab5a (DEFAULT_TABLE table=default.Table_fde9bb8a8be244ed9462ed7367173e96 [id=4f930e9fc544463fad7425810a0f6337]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.226500 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 92dd966c6bcc47d187cfdc9fcad7848a (DEFAULT_TABLE table=default.Table_b7298917793f498a8273bb74124d7807 [id=27394e7f480042199cd9397185b970d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.226766 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 8e2b23acac89427caef937831c4d2737 (DEFAULT_TABLE table=default.Table_d8455ca7030842aa80ba3d8da6c5a632 [id=5e643d181cc84f3eb8acf5449332100a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.237107 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146d6f13b9ef47f29711a102791bab5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.237311 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92dd966c6bcc47d187cfdc9fcad7848a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.237325 3841 leader_election.cc:290] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:44.238036 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2b23acac89427caef937831c4d2737. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.247723 3928 tablet.cc:1722] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.232460 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 92dd966c6bcc47d187cfdc9fcad7848a (DEFAULT_TABLE table=default.Table_b7298917793f498a8273bb74124d7807 [id=27394e7f480042199cd9397185b970d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.232378 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 8e2b23acac89427caef937831c4d2737 (DEFAULT_TABLE table=default.Table_d8455ca7030842aa80ba3d8da6c5a632 [id=5e643d181cc84f3eb8acf5449332100a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.248886 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2b23acac89427caef937831c4d2737. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.248306 3925 tablet.cc:1717] T de8799a5af2b42cc92ccc0195cff98e7 P 7fef01f4f2b843538ea245a09dd087a8: Skipping requested alter to schema version 3, tablet already version 3
I20251026 02:56:44.232443 2242 tablet_service.cc:1505] Processing CreateTablet for tablet f86766e836be46c3978917beb2797964 (DEFAULT_TABLE table=default.Table_7941853c36584b5591a75bf490bdb543 [id=f7002d452201483e829922a1b22277d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.251112 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f86766e836be46c3978917beb2797964. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:44.251772 2194 consensus_peers.cc:597] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:44.238524 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62d30d79a6e4432a8dbe40a104e87fa0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:44.247166 3890 raft_consensus.cc:493] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.252665 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92dd966c6bcc47d187cfdc9fcad7848a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:44.252629 2053 leader_election.cc:343] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.252362 3890 raft_consensus.cc:515] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.252959 3929 tablet.cc:1722] T 8862cd48a5744c99b32f365cb77fa7c4 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.253741 2377 raft_consensus.cc:2272] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.254109 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e483c52e514f1da54335479f59d68e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:44.254310 2110 raft_consensus.cc:2468] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:44.254657 2195 leader_election.cc:304] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:44.254906 3877 raft_consensus.cc:2804] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:44.254984 3877 raft_consensus.cc:493] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:44.255059 3877 raft_consensus.cc:3060] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.255471 2377 ts_tablet_manager.cc:1916] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.259164 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e483c52e514f1da54335479f59d68e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:44.257589 3877 raft_consensus.cc:515] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.260597 3877 leader_election.cc:290] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:44.258302 3890 leader_election.cc:290] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:44.259634 2194 leader_election.cc:343] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.261092 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e483c52e514f1da54335479f59d68e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:44.261268 2110 raft_consensus.cc:3060] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.265375 2110 raft_consensus.cc:2468] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:44.265832 2195 leader_election.cc:304] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:44.266032 2372 ts_tablet_manager.cc:1929] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.266165 3890 raft_consensus.cc:2804] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:44.266387 3890 raft_consensus.cc:697] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:44.266680 3890 consensus_queue.cc:237] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.269742 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e483c52e514f1da54335479f59d68e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:44.269868 2372 log.cc:1199] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/b1ca5b8379c94deab1130bfe8bdc87b0
I20251026 02:56:44.270406 2372 ts_tablet_manager.cc:1950] T b1ca5b8379c94deab1130bfe8bdc87b0 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:44.272431 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 146d6f13b9ef47f29711a102791bab5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:44.273303 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 759) took 6522 ms. Trace:
I20251026 02:56:44.274099 2372 rpcz_store.cc:276] 1026 02:56:37.750782 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.750842 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:44.076801 (+6325959us) ts_tablet_manager.cc:1151] Deleting tablet b1ca5b8379c94deab1130bfe8bdc87b0
1026 02:56:44.076809 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.137524 (+ 60715us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.265903 (+128379us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.273286 (+ 7383us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6245816}
W20251026 02:56:44.275504 2194 leader_election.cc:343] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.276463 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet b1ca5b8379c94deab1130bfe8bdc87b0 (table default.Table_85c523c971b34e68bed4afa5d1aac450 [id=32c06e4782a8441c8d1614e81277b52e]) successfully deleted
I20251026 02:56:44.276994 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 762) took 6460 ms. Trace:
I20251026 02:56:44.277643 2372 rpcz_store.cc:276] 1026 02:56:37.817009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.817088 (+ 79us) service_pool.cc:225] Handling call
1026 02:56:44.276861 (+6459773us) ts_tablet_manager.cc:1151] Deleting tablet 9a5670a9e10c4a9db64881e7e14e839e
1026 02:56:44.276871 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.276975 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6435984}
W20251026 02:56:44.280042 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9a5670a9e10c4a9db64881e7e14e839e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a5670a9e10c4a9db64881e7e14e839e already in progress: creating tablet
I20251026 02:56:44.280725 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 146d6f13b9ef47f29711a102791bab5a (DEFAULT_TABLE table=default.Table_fde9bb8a8be244ed9462ed7367173e96 [id=4f930e9fc544463fad7425810a0f6337]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.281051 2372 tablet_replica.cc:333] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:44.281279 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146d6f13b9ef47f29711a102791bab5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.281901 2372 raft_consensus.cc:2243] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:44.285156 2372 raft_consensus.cc:2272] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.287988 2372 ts_tablet_manager.cc:1916] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.291291 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62d30d79a6e4432a8dbe40a104e87fa0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:44.291894 2053 leader_election.cc:343] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.294503 2053 leader_election.cc:304] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:44.294266 3866 tablet.cc:1722] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.295233 3841 raft_consensus.cc:2749] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:44.307945 1980 raft_consensus.cc:2468] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 2.
I20251026 02:56:44.308943 3866 tablet.cc:1722] T 8adf5c6970be46ed8434fd28404b2166 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.313449 2194 leader_election.cc:304] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:44.313705 3893 raft_consensus.cc:2804] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 02:56:44.313834 3893 raft_consensus.cc:697] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 2 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:44.314198 3893 consensus_queue.cc:237] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:44.325490 2377 ts_tablet_manager.cc:1929] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.325693 2377 log.cc:1199] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/aa0aa6cdd0de4b0a91f553c2341e9c01
I20251026 02:56:44.326084 2377 ts_tablet_manager.cc:1950] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:44.330102 2376 ts_tablet_manager.cc:1929] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251026 02:56:44.330878 2376 log.cc:1199] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/47037db3602246a2ab6224557286162f
I20251026 02:56:44.333009 2376 ts_tablet_manager.cc:1950] T 47037db3602246a2ab6224557286162f P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:44.334816 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 780) took 4725 ms. Trace:
I20251026 02:56:44.336268 2378 tablet_bootstrap.cc:492] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:44.336447 2378 ts_tablet_manager.cc:1403] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.160s user 0.005s sys 0.003s
I20251026 02:56:44.335453 2376 rpcz_store.cc:276] 1026 02:56:39.609040 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.609092 (+ 52us) service_pool.cc:225] Handling call
1026 02:56:44.152962 (+4543870us) ts_tablet_manager.cc:1151] Deleting tablet 47037db3602246a2ab6224557286162f
1026 02:56:44.152972 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.309490 (+156518us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.329526 (+ 20036us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.334639 (+ 5113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4543716}
I20251026 02:56:44.337179 2378 raft_consensus.cc:359] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.337380 2378 raft_consensus.cc:385] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:44.337468 2378 raft_consensus.cc:740] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.337934 2378 consensus_queue.cc:260] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.338635 2378 ts_tablet_manager.cc:1434] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:44.343218 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 765) took 6205 ms. Trace:
I20251026 02:56:44.343370 2377 rpcz_store.cc:276] 1026 02:56:38.137742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.138131 (+ 389us) service_pool.cc:225] Handling call
1026 02:56:44.215640 (+6077509us) ts_tablet_manager.cc:1151] Deleting tablet aa0aa6cdd0de4b0a91f553c2341e9c01
1026 02:56:44.215653 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.280806 (+ 65153us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.325362 (+ 44556us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.343193 (+ 17831us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6077275}
I20251026 02:56:44.343858 2377 tablet_replica.cc:333] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:44.344079 2377 raft_consensus.cc:2243] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:44.344332 2377 pending_rounds.cc:70] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:44.344437 2377 pending_rounds.cc:77] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888564626075648 op_type: NO_OP noop_request { }
I20251026 02:56:44.344566 2377 raft_consensus.cc:2889] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:44.344678 2377 raft_consensus.cc:2272] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.345774 2372 ts_tablet_manager.cc:1929] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.346629 2377 ts_tablet_manager.cc:1916] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.361173 2372 log.cc:1199] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/1370b9ad917344c093e5da3d0e9b1cf0
I20251026 02:56:44.361476 2377 ts_tablet_manager.cc:1929] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.361668 2372 ts_tablet_manager.cc:1950] T 1370b9ad917344c093e5da3d0e9b1cf0 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:44.362136 2377 log.cc:1199] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/fe48472efa444bf580e4074c96a9de92
I20251026 02:56:44.362679 2377 ts_tablet_manager.cc:1950] T fe48472efa444bf580e4074c96a9de92 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:44.363435 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 766) took 6109 ms. Trace:
I20251026 02:56:44.363893 2377 rpcz_store.cc:276] 1026 02:56:38.253845 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.255381 (+ 1536us) service_pool.cc:225] Handling call
1026 02:56:44.343807 (+6088426us) ts_tablet_manager.cc:1151] Deleting tablet fe48472efa444bf580e4074c96a9de92
1026 02:56:44.343818 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.349790 (+ 5972us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.361351 (+ 11561us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.363418 (+ 2067us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6015339}
I20251026 02:56:44.364641 2377 tablet_replica.cc:333] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:44.365211 2377 raft_consensus.cc:2243] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:44.365671 2377 raft_consensus.cc:2272] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.367774 2377 ts_tablet_manager.cc:1916] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.372042 2165 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:44.373132 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 765) took 6522 ms. Trace:
I20251026 02:56:44.375540 2372 rpcz_store.cc:276] 1026 02:56:37.850193 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.852269 (+ 2076us) service_pool.cc:225] Handling call
1026 02:56:44.281001 (+6428732us) ts_tablet_manager.cc:1151] Deleting tablet 1370b9ad917344c093e5da3d0e9b1cf0
1026 02:56:44.281011 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.298304 (+ 17293us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.345660 (+ 47356us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.373107 (+ 27447us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6428555}
I20251026 02:56:44.377864 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 766) took 6507 ms. Trace:
I20251026 02:56:44.357139 3841 raft_consensus.cc:493] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.377995 2372 rpcz_store.cc:276] 1026 02:56:37.870314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.870625 (+ 311us) service_pool.cc:225] Handling call
1026 02:56:44.377723 (+6507098us) ts_tablet_manager.cc:1151] Deleting tablet b1b6613742a24c21a1db6ebe858a846f
1026 02:56:44.377736 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.377848 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6506869}
I20251026 02:56:44.378119 3841 raft_consensus.cc:515] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.378564 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 768) took 6445 ms. Trace:
I20251026 02:56:44.378629 2372 rpcz_store.cc:276] 1026 02:56:37.932626 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.943600 (+ 10974us) service_pool.cc:225] Handling call
1026 02:56:44.378462 (+6434862us) ts_tablet_manager.cc:1151] Deleting tablet 3aa7a90a29ec430286786bf8aa9d4c02
1026 02:56:44.378471 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.378550 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6434658}
I20251026 02:56:44.378736 3841 leader_election.cc:290] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:44.378966 2372 tablet_replica.cc:333] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:44.379158 2372 raft_consensus.cc:2243] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:44.379364 2372 raft_consensus.cc:2272] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.379618 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c1e63ac8c5645f3ba1578792c0a0dae" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:44.380537 2053 leader_election.cc:343] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.380542 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c1e63ac8c5645f3ba1578792c0a0dae" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:44.355949 2375 raft_consensus.cc:385] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251026 02:56:44.381103 2053 leader_election.cc:343] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.381186 2053 leader_election.cc:304] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:44.381690 3841 raft_consensus.cc:2749] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:44.380823 2375 raft_consensus.cc:740] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.382273 2375 consensus_queue.cc:260] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.384478 2375 ts_tablet_manager.cc:1434] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.250s user 0.014s sys 0.003s
I20251026 02:56:44.388499 2377 ts_tablet_manager.cc:1929] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.390533 2375 tablet_bootstrap.cc:492] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:44.392325 2377 log.cc:1199] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/d90e62ae86b149b291dc29cf7442c6bc
I20251026 02:56:44.392875 2377 ts_tablet_manager.cc:1950] T d90e62ae86b149b291dc29cf7442c6bc P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:44.393625 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 767) took 6072 ms. Trace:
I20251026 02:56:44.394115 2377 rpcz_store.cc:276] 1026 02:56:38.321667 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.321724 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:44.364595 (+6042871us) ts_tablet_manager.cc:1151] Deleting tablet d90e62ae86b149b291dc29cf7442c6bc
1026 02:56:44.364603 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.382045 (+ 17442us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.388384 (+ 6339us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.393609 (+ 5225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6024277}
I20251026 02:56:44.396595 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 768) took 5765 ms. Trace:
I20251026 02:56:44.398430 3893 consensus_queue.cc:1048] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:44.399806 2375 tablet_bootstrap.cc:654] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.398216 2372 ts_tablet_manager.cc:1916] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.411744 3841 raft_consensus.cc:493] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.411906 3841 raft_consensus.cc:515] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.412524 3841 leader_election.cc:290] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:44.359263 1657 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 8862cd48a5744c99b32f365cb77fa7c4 (table default.Table_dabd227b99964351870151b7731b8175 [id=604570227c014f66aeae89d93edaaafe]). Expected version 1 got 0
I20251026 02:56:44.377511 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 1370b9ad917344c093e5da3d0e9b1cf0 (table default.Table_765fd8cb95b14c77976811056434b451 [id=555da09ead8f4786944938514a53b4e9]) successfully deleted
W20251026 02:56:44.413228 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3aa7a90a29ec430286786bf8aa9d4c02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3aa7a90a29ec430286786bf8aa9d4c02 already in progress: creating tablet
I20251026 02:56:44.413388 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b5e61afe90e4bc4a2897de21e0b3919" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:44.413627 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b5e61afe90e4bc4a2897de21e0b3919" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:44.414672 2053 leader_election.cc:343] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:44.415109 2053 leader_election.cc:343] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.415211 2053 leader_election.cc:304] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:44.415619 3841 raft_consensus.cc:2749] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:44.418253 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet aa0aa6cdd0de4b0a91f553c2341e9c01 (table default.Table_1a48da091e984e09aaffb9caed79f2c1 [id=1b3988cec795459f8666476a9f979f3a]) successfully deleted
I20251026 02:56:44.416996 2377 rpcz_store.cc:276] 1026 02:56:38.630730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.631044 (+ 314us) service_pool.cc:225] Handling call
1026 02:56:44.396420 (+5765376us) ts_tablet_manager.cc:1151] Deleting tablet 0cf76ef8f6af445eb0ce15c8d79136c5
1026 02:56:44.396430 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.396579 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5765137}
I20251026 02:56:44.418613 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet fe48472efa444bf580e4074c96a9de92 (table default.Table_fc676ef48d7e4c7bbb2fac99cdf17176 [id=7e3d19f1d75d4ebb94c220327ff8abcb]) successfully deleted
I20251026 02:56:44.418613 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 770) took 5778 ms. Trace:
I20251026 02:56:44.418708 2377 rpcz_store.cc:276] 1026 02:56:38.640178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.640244 (+ 66us) service_pool.cc:225] Handling call
1026 02:56:44.418460 (+5778216us) ts_tablet_manager.cc:1151] Deleting tablet 2f1872ef9f244a388d25a8085dd53f82
1026 02:56:44.418471 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.418593 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5753851}
I20251026 02:56:44.418818 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet d90e62ae86b149b291dc29cf7442c6bc (table default.Table_e2a5e05b70f841cfb0065041b39e89cb [id=e1c065cb2f704594873006d80a406df2]) successfully deleted
W20251026 02:56:44.419003 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0cf76ef8f6af445eb0ce15c8d79136c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cf76ef8f6af445eb0ce15c8d79136c5 already in progress: creating tablet
I20251026 02:56:44.419023 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 769) took 5782 ms. Trace:
I20251026 02:56:44.419202 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 47037db3602246a2ab6224557286162f (table default.Table_09ca38422af543f89eab26b4117536dc [id=12a8deac66914c6da970cd7f975b5b59]) successfully deleted
I20251026 02:56:44.419080 2377 rpcz_store.cc:276] 1026 02:56:38.636596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.641114 (+ 4518us) service_pool.cc:225] Handling call
1026 02:56:44.418942 (+5777828us) ts_tablet_manager.cc:1151] Deleting tablet 86d8e1e21c5f440d8baf76c357d01ad2
1026 02:56:44.418951 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.419006 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5752424}
W20251026 02:56:44.419410 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 2f1872ef9f244a388d25a8085dd53f82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f1872ef9f244a388d25a8085dd53f82 already in progress: creating tablet
I20251026 02:56:44.419540 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 771) took 5768 ms. Trace:
W20251026 02:56:44.419639 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 86d8e1e21c5f440d8baf76c357d01ad2 because the tablet was not found. No further retry: Not found: Tablet not found: 86d8e1e21c5f440d8baf76c357d01ad2
I20251026 02:56:44.419610 2377 rpcz_store.cc:276] 1026 02:56:38.650741 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.650809 (+ 68us) service_pool.cc:225] Handling call
1026 02:56:44.419441 (+5768632us) ts_tablet_manager.cc:1151] Deleting tablet 3ac38e6257e74367b06b15fc49d2af81
1026 02:56:44.419449 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.419525 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5752809}
I20251026 02:56:44.419915 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 772) took 5769 ms. Trace:
I20251026 02:56:44.419991 2377 rpcz_store.cc:276] 1026 02:56:38.650851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.650896 (+ 45us) service_pool.cc:225] Handling call
1026 02:56:44.419820 (+5768924us) ts_tablet_manager.cc:1151] Deleting tablet 62d30d79a6e4432a8dbe40a104e87fa0
1026 02:56:44.419830 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.419901 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5753088}
I20251026 02:56:44.420472 2377 tablet_replica.cc:333] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:44.420717 2377 raft_consensus.cc:2243] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:44.421203 2377 raft_consensus.cc:2272] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.423084 2377 ts_tablet_manager.cc:1916] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:44.423440 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 62d30d79a6e4432a8dbe40a104e87fa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62d30d79a6e4432a8dbe40a104e87fa0 already in progress: creating tablet
I20251026 02:56:44.426420 3372 raft_consensus.cc:493] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.426620 3372 raft_consensus.cc:515] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.427438 3372 leader_election.cc:290] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:44.427910 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c6e3e47b3e7421e945cfbe671b06abe" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:44.428313 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c6e3e47b3e7421e945cfbe671b06abe" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:44.428741 1915 leader_election.cc:343] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: Tablet not RUNNING: STOPPED
I20251026 02:56:44.429179 1916 leader_election.cc:304] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:44.429679 3372 raft_consensus.cc:2749] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:44.437484 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 783) took 4744 ms. Trace:
I20251026 02:56:44.437592 2376 rpcz_store.cc:276] 1026 02:56:39.692969 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.693282 (+ 313us) service_pool.cc:225] Handling call
1026 02:56:44.337899 (+4644617us) ts_tablet_manager.cc:1151] Deleting tablet 112181b9207e448f959f8311ad95a02b
1026 02:56:44.437298 (+ 99399us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.437434 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4599472}
I20251026 02:56:44.438580 2378 tablet_bootstrap.cc:492] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:44.438797 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 784) took 4743 ms. Trace:
I20251026 02:56:44.438863 2376 rpcz_store.cc:276] 1026 02:56:39.694882 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.695004 (+ 122us) service_pool.cc:225] Handling call
1026 02:56:44.437837 (+4742833us) ts_tablet_manager.cc:1151] Deleting tablet c2d6691bb4cf4a8292b0c082e54a91a5
1026 02:56:44.438681 (+ 844us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.438783 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4696211}
I20251026 02:56:44.439119 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 788) took 4739 ms. Trace:
I20251026 02:56:44.439167 2376 rpcz_store.cc:276] 1026 02:56:39.700133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.700191 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:44.439051 (+4738860us) ts_tablet_manager.cc:1151] Deleting tablet 144b957c69e64085a6f78050dea426b4
1026 02:56:44.439058 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.439109 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4690701}
I20251026 02:56:44.439363 2376 tablet_replica.cc:333] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:44.439527 2376 raft_consensus.cc:2243] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:44.439670 2376 raft_consensus.cc:2272] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.441314 2376 ts_tablet_manager.cc:1916] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.443732 2372 ts_tablet_manager.cc:1929] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.444548 3888 consensus_queue.cc:1048] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:44.446426 3372 raft_consensus.cc:493] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.446583 3372 raft_consensus.cc:515] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:44.446993 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 112181b9207e448f959f8311ad95a02b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 112181b9207e448f959f8311ad95a02b already in progress: creating tablet
I20251026 02:56:44.447234 3372 leader_election.cc:290] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:44.447489 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet c2d6691bb4cf4a8292b0c082e54a91a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2d6691bb4cf4a8292b0c082e54a91a5 already in progress: creating tablet
W20251026 02:56:44.447871 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 144b957c69e64085a6f78050dea426b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 144b957c69e64085a6f78050dea426b4 already in progress: creating tablet
W20251026 02:56:44.448513 1916 leader_election.cc:343] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ed836bb7519e4b0290e6547fa57128bc
I20251026 02:56:44.449021 2372 log.cc:1199] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/882fe81da21344ed99ea735fa79f73c0
I20251026 02:56:44.449656 2372 ts_tablet_manager.cc:1950] T 882fe81da21344ed99ea735fa79f73c0 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:44.450716 1915 leader_election.cc:343] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: ed836bb7519e4b0290e6547fa57128bc
I20251026 02:56:44.450816 1915 leader_election.cc:304] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:44.451459 3372 raft_consensus.cc:2749] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:44.452612 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 769) took 6494 ms. Trace:
I20251026 02:56:44.452701 2372 rpcz_store.cc:276] 1026 02:56:37.958067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:37.958144 (+ 77us) service_pool.cc:225] Handling call
1026 02:56:44.378925 (+6420781us) ts_tablet_manager.cc:1151] Deleting tablet 882fe81da21344ed99ea735fa79f73c0
1026 02:56:44.378933 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.416312 (+ 37379us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.443621 (+ 27309us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.452592 (+ 8971us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6420574}
I20251026 02:56:44.453148 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 882fe81da21344ed99ea735fa79f73c0 (table default.Table_18e8723148dd46e6a56e5716a15d01bc [id=156e8ddc92b1471e8dce064bcb23306b]) successfully deleted
W20251026 02:56:44.454146 2194 consensus_peers.cc:597] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:44.455763 2377 ts_tablet_manager.cc:1929] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.456372 2372 tablet_replica.cc:333] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:44.456506 2377 log.cc:1199] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/5c6e3e47b3e7421e945cfbe671b06abe
I20251026 02:56:44.456797 2372 raft_consensus.cc:2243] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:44.456911 2377 ts_tablet_manager.cc:1950] T 5c6e3e47b3e7421e945cfbe671b06abe P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:44.457222 2372 raft_consensus.cc:2272] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.460847 2372 ts_tablet_manager.cc:1916] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.465090 1975 raft_consensus.cc:3060] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.471616 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_35a36ec0f72d4af79182e095d6788ca4" } new_table_name: "default.Table_5ba68a14fa57401386635f740119713f" modify_external_catalogs: true
I20251026 02:56:44.481051 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 773) took 5830 ms. Trace:
I20251026 02:56:44.482594 2377 rpcz_store.cc:276] 1026 02:56:38.650967 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.651020 (+ 53us) service_pool.cc:225] Handling call
1026 02:56:44.420423 (+5769403us) ts_tablet_manager.cc:1151] Deleting tablet 5c6e3e47b3e7421e945cfbe671b06abe
1026 02:56:44.420432 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.429919 (+ 9487us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.455638 (+ 25719us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.481025 (+ 25387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5753569}
W20251026 02:56:44.483385 1629 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 237265us
I20251026 02:56:44.483951 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 774) took 5826 ms. Trace:
I20251026 02:56:44.485570 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 5c6e3e47b3e7421e945cfbe671b06abe (table default.Table_653a3f85c24140cdb62a3e9f3ea95fd4 [id=a305da1e3a694b32aff05dbd6c58a6cb]) successfully deleted
I20251026 02:56:44.485725 3893 raft_consensus.cc:493] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.485875 3893 raft_consensus.cc:515] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.486518 3893 leader_election.cc:290] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:44.487062 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f719c720cc474181a63786a3e248ab2c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:44.487639 2194 leader_election.cc:343] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:44.487995 2195 leader_election.cc:343] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: f719c720cc474181a63786a3e248ab2c
I20251026 02:56:44.488088 2195 leader_election.cc:304] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:44.488683 3893 raft_consensus.cc:2749] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:44.484045 2377 rpcz_store.cc:276] 1026 02:56:38.657159 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.659407 (+ 2248us) service_pool.cc:225] Handling call
1026 02:56:44.483802 (+5824395us) ts_tablet_manager.cc:1151] Deleting tablet 863aef239cd34a358a71ddfdba6fb72a
1026 02:56:44.483815 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.483930 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5816862}
I20251026 02:56:44.490976 3841 consensus_queue.cc:1048] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:44.491261 3496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=408 overwritten=1 applied=407 ignored=0} inserts{seen=383 ignored=0} mutations{seen=462 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 02:56:44.492053 3496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: Bootstrap complete.
W20251026 02:56:44.492264 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 863aef239cd34a358a71ddfdba6fb72a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 863aef239cd34a358a71ddfdba6fb72a already in progress: creating tablet
I20251026 02:56:44.493945 2372 ts_tablet_manager.cc:1929] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.495047 2372 log.cc:1199] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/aa0aa6cdd0de4b0a91f553c2341e9c01
I20251026 02:56:44.495622 2372 ts_tablet_manager.cc:1950] T aa0aa6cdd0de4b0a91f553c2341e9c01 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:44.497081 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 770) took 6453 ms. Trace:
I20251026 02:56:44.496126 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 800) took 5487 ms. Trace:
I20251026 02:56:44.497700 2377 rpcz_store.cc:276] 1026 02:56:39.008656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.009149 (+ 493us) service_pool.cc:225] Handling call
1026 02:56:44.493812 (+5484663us) ts_tablet_manager.cc:1151] Deleting tablet 02861f399f4d488586f279c01eebcdd1
1026 02:56:44.493824 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.495712 (+ 1888us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5484274}
W20251026 02:56:44.501286 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 02861f399f4d488586f279c01eebcdd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02861f399f4d488586f279c01eebcdd1 already in progress: creating tablet
I20251026 02:56:44.504360 2372 rpcz_store.cc:276] 1026 02:56:38.043898 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.044153 (+ 255us) service_pool.cc:225] Handling call
1026 02:56:44.456322 (+6412169us) ts_tablet_manager.cc:1151] Deleting tablet aa0aa6cdd0de4b0a91f553c2341e9c01
1026 02:56:44.456331 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.465310 (+ 8979us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.493798 (+ 28488us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.497053 (+ 3255us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6411911}
I20251026 02:56:44.504583 3893 raft_consensus.cc:493] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.504738 3893 raft_consensus.cc:515] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.505252 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet aa0aa6cdd0de4b0a91f553c2341e9c01 (table default.Table_1a48da091e984e09aaffb9caed79f2c1 [id=1b3988cec795459f8666476a9f979f3a]) successfully deleted
I20251026 02:56:44.505347 3893 leader_election.cc:290] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:44.505635 2372 tablet_replica.cc:333] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:44.506062 2372 raft_consensus.cc:2243] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:44.505239 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 799) took 5496 ms. Trace:
I20251026 02:56:44.506485 2372 raft_consensus.cc:2272] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.506615 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7abcdac4a70b48ad9fbba1bc8a60283b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:44.507149 2110 raft_consensus.cc:2468] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:44.507279 2372 ts_tablet_manager.cc:1916] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.507891 2195 leader_election.cc:304] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:44.508270 3890 raft_consensus.cc:2804] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:44.508347 3890 raft_consensus.cc:493] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:44.508404 3890 raft_consensus.cc:3060] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.506634 2377 rpcz_store.cc:276] 1026 02:56:39.008532 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.009373 (+ 841us) service_pool.cc:225] Handling call
1026 02:56:44.503187 (+5493814us) ts_tablet_manager.cc:1151] Deleting tablet 46547883cfe041f3954aa4128fba1d8c
1026 02:56:44.503206 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.505024 (+ 1818us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5493050}
W20251026 02:56:44.509932 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 46547883cfe041f3954aa4128fba1d8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46547883cfe041f3954aa4128fba1d8c already in progress: creating tablet
I20251026 02:56:44.511041 3890 raft_consensus.cc:515] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.511621 3890 leader_election.cc:290] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:44.512596 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7abcdac4a70b48ad9fbba1bc8a60283b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:44.513042 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7abcdac4a70b48ad9fbba1bc8a60283b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:44.513211 2110 raft_consensus.cc:3060] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.513492 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 801) took 5483 ms. Trace:
W20251026 02:56:44.516587 2194 leader_election.cc:343] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.514319 2377 rpcz_store.cc:276] 1026 02:56:39.029598 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.262593 (+232995us) service_pool.cc:225] Handling call
1026 02:56:44.511598 (+5249005us) ts_tablet_manager.cc:1151] Deleting tablet 3e424a9c89e64b1c98ada17fa5c31e9e
1026 02:56:44.511613 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.513291 (+ 1678us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5248364}
I20251026 02:56:44.516180 2110 raft_consensus.cc:2468] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:44.518748 2195 leader_election.cc:304] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:44.518941 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3e424a9c89e64b1c98ada17fa5c31e9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e424a9c89e64b1c98ada17fa5c31e9e already in progress: creating tablet
I20251026 02:56:44.519341 3893 raft_consensus.cc:2804] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:44.519562 3893 raft_consensus.cc:697] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:44.519801 3893 consensus_queue.cc:237] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.521809 1945 tablet_service.cc:1505] Processing CreateTablet for tablet cffa8c7cf71a4d449803e163b415f473 (DEFAULT_TABLE table=default.Table_34885bd9ba024c96a4bea0c2fedcfb4b [id=94a166ea5a59425797bb4af071a97abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.522550 2082 tablet_service.cc:1505] Processing CreateTablet for tablet cffa8c7cf71a4d449803e163b415f473 (DEFAULT_TABLE table=default.Table_34885bd9ba024c96a4bea0c2fedcfb4b [id=94a166ea5a59425797bb4af071a97abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.522943 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cffa8c7cf71a4d449803e163b415f473. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.524827 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 818) took 5126 ms. Trace:
I20251026 02:56:44.525653 2377 rpcz_store.cc:276] 1026 02:56:39.397980 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.398028 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:44.523244 (+5125216us) ts_tablet_manager.cc:1151] Deleting tablet 7053d454def04264a956aa75ceaad5c9
1026 02:56:44.523264 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.524609 (+ 1345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5124254}
I20251026 02:56:44.528978 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 7aa3b6d88e914dea90ec55d43f7c1e1b (DEFAULT_TABLE table=default.Table_a53aedec98894542aa3c419b2295c67d [id=b40bbb71e5a244b0923d72e1ce617680]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.529583 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 7aa3b6d88e914dea90ec55d43f7c1e1b (DEFAULT_TABLE table=default.Table_a53aedec98894542aa3c419b2295c67d [id=b40bbb71e5a244b0923d72e1ce617680]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.529913 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa3b6d88e914dea90ec55d43f7c1e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.529433 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 817) took 5131 ms. Trace:
I20251026 02:56:44.531250 2372 ts_tablet_manager.cc:1929] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:44.534392 2229 tablet_service.cc:1505] Processing CreateTablet for tablet cffa8c7cf71a4d449803e163b415f473 (DEFAULT_TABLE table=default.Table_34885bd9ba024c96a4bea0c2fedcfb4b [id=94a166ea5a59425797bb4af071a97abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.534790 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cffa8c7cf71a4d449803e163b415f473. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.532042 2375 tablet_bootstrap.cc:492] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:44.535470 2375 ts_tablet_manager.cc:1403] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.145s user 0.009s sys 0.013s
I20251026 02:56:44.536105 2375 raft_consensus.cc:359] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.536391 2375 raft_consensus.cc:385] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:44.536893 2375 raft_consensus.cc:740] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.537145 2375 consensus_queue.cc:260] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.537685 2375 ts_tablet_manager.cc:1434] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:44.537993 2375 tablet_bootstrap.cc:492] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:44.539448 2375 tablet_bootstrap.cc:654] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.539501 2378 tablet_bootstrap.cc:654] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.540100 2372 log.cc:1199] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/fe48472efa444bf580e4074c96a9de92
I20251026 02:56:44.540581 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 7aa3b6d88e914dea90ec55d43f7c1e1b (DEFAULT_TABLE table=default.Table_a53aedec98894542aa3c419b2295c67d [id=b40bbb71e5a244b0923d72e1ce617680]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.540974 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa3b6d88e914dea90ec55d43f7c1e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.541241 2372 ts_tablet_manager.cc:1950] T fe48472efa444bf580e4074c96a9de92 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:44.541992 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 771) took 6336 ms. Trace:
I20251026 02:56:44.542630 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_74606f0d7edf4d7281d9d30c1e6f052f" } new_table_name: "default.Table_f78aa520baff4e07be8f8a125262a3bd" modify_external_catalogs: true
I20251026 02:56:44.542541 3496 raft_consensus.cc:359] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:44.543676 2372 rpcz_store.cc:276] 1026 02:56:38.205249 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.205313 (+ 64us) service_pool.cc:225] Handling call
1026 02:56:44.505577 (+6300264us) ts_tablet_manager.cc:1151] Deleting tablet fe48472efa444bf580e4074c96a9de92
1026 02:56:44.505590 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.510589 (+ 4999us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.531146 (+ 20557us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.541972 (+ 10826us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6300048}
I20251026 02:56:44.544270 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet fe48472efa444bf580e4074c96a9de92 (table default.Table_fc676ef48d7e4c7bbb2fac99cdf17176 [id=7e3d19f1d75d4ebb94c220327ff8abcb]) successfully deleted
I20251026 02:56:44.544622 2372 tablet_replica.cc:333] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:44.544847 2372 raft_consensus.cc:2243] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:44.545015 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa3b6d88e914dea90ec55d43f7c1e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.545034 2372 raft_consensus.cc:2272] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.545449 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fe71a1f386b943db95b6684602e55c22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:44.545571 2372 ts_tablet_manager.cc:1916] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:44.546036 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe71a1f386b943db95b6684602e55c22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:44.543514 3496 raft_consensus.cc:740] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751b719b75ce4bc2a88a19091230ecb7, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.548439 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cffa8c7cf71a4d449803e163b415f473. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.551085 3496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 407, Last appended: 3.407, Last appended by leader: 407, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:44.564781 2381 tablet_bootstrap.cc:492] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:44.565017 2381 ts_tablet_manager.cc:1403] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.460s user 0.034s sys 0.032s
I20251026 02:56:44.565861 2381 raft_consensus.cc:359] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.568905 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_51410f2fee7b4ac595de28ae74467f89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:44.569633 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51410f2fee7b4ac595de28ae74467f89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:44.570664 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7053d454def04264a956aa75ceaad5c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7053d454def04264a956aa75ceaad5c9 already in progress: creating tablet
I20251026 02:56:44.571161 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7abcdac4a70b48ad9fbba1bc8a60283b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:44.583921 2372 ts_tablet_manager.cc:1929] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.585052 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ce57d8a514194198ad913d3eb35269b9" } new_table_name: "default.Table_1d748ce60f124393b4f4ad96a131fbaf" modify_external_catalogs: true
I20251026 02:56:44.585808 2372 log.cc:1199] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/d90e62ae86b149b291dc29cf7442c6bc
I20251026 02:56:44.586557 2372 ts_tablet_manager.cc:1950] T d90e62ae86b149b291dc29cf7442c6bc P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:44.587946 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 772) took 6311 ms. Trace:
I20251026 02:56:44.589015 2372 rpcz_store.cc:276] 1026 02:56:38.276425 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.276490 (+ 65us) service_pool.cc:225] Handling call
1026 02:56:44.544569 (+6268079us) ts_tablet_manager.cc:1151] Deleting tablet d90e62ae86b149b291dc29cf7442c6bc
1026 02:56:44.544583 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.563099 (+ 18516us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.583765 (+ 20666us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.587926 (+ 4161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6267859}
I20251026 02:56:44.589596 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 8e3e3e8eed68443a895e2ac04b217de7 (DEFAULT_TABLE table=default.Table_3c1d783b88b0493f8aac8545f4cb3049 [id=16bc2124a2314e55bcb20616ac7879e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.590083 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3e3e8eed68443a895e2ac04b217de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.590121 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet d90e62ae86b149b291dc29cf7442c6bc (table default.Table_e2a5e05b70f841cfb0065041b39e89cb [id=e1c065cb2f704594873006d80a406df2]) successfully deleted
I20251026 02:56:44.590582 2372 tablet_replica.cc:333] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:44.591348 2372 raft_consensus.cc:2243] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:44.591856 1650 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 1a1b8c440061429caea0f0803f110742 (table default.Table_38f23e9599964426a78e8e11b9fa08bb [id=43cb7f538f824ae38059645230cc7e94], current schema version=5) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1a1b8c440061429caea0f0803f110742 consensus state has no leader
I20251026 02:56:44.592077 2372 raft_consensus.cc:2272] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.593123 2372 ts_tablet_manager.cc:1916] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:44.572696 2194 leader_election.cc:343] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.573170 3904 tablet.cc:1722] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.595108 3866 tablet.cc:1722] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.595677 3929 tablet.cc:1722] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:44.596640 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 8e3e3e8eed68443a895e2ac04b217de7 (DEFAULT_TABLE table=default.Table_3c1d783b88b0493f8aac8545f4cb3049 [id=16bc2124a2314e55bcb20616ac7879e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.597085 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3e3e8eed68443a895e2ac04b217de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.593097 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 8e3e3e8eed68443a895e2ac04b217de7 (DEFAULT_TABLE table=default.Table_3c1d783b88b0493f8aac8545f4cb3049 [id=16bc2124a2314e55bcb20616ac7879e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.599560 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3e3e8eed68443a895e2ac04b217de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.530092 2377 rpcz_store.cc:276] 1026 02:56:39.397884 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.400567 (+ 2683us) service_pool.cc:225] Handling call
1026 02:56:44.528002 (+5127435us) ts_tablet_manager.cc:1151] Deleting tablet b819b43894574dc9ad52b4081e46628e
1026 02:56:44.528122 (+ 120us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.529289 (+ 1167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5127020}
I20251026 02:56:44.603590 2377 tablet_replica.cc:333] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:44.603804 2377 raft_consensus.cc:2243] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 2 LEADER]: Raft consensus shutting down.
I20251026 02:56:44.604163 2377 raft_consensus.cc:2272] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:44.605082 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet b819b43894574dc9ad52b4081e46628e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b819b43894574dc9ad52b4081e46628e already in progress: creating tablet
I20251026 02:56:44.608901 2377 ts_tablet_manager.cc:1916] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.612295 2378 tablet_bootstrap.cc:492] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:44.612511 2378 ts_tablet_manager.cc:1403] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.174s user 0.006s sys 0.006s
I20251026 02:56:44.613317 2378 raft_consensus.cc:359] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.613533 2378 raft_consensus.cc:385] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:44.613618 2378 raft_consensus.cc:740] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.613862 2378 consensus_queue.cc:260] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.614504 2378 ts_tablet_manager.cc:1434] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:44.614634 2381 raft_consensus.cc:385] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:44.614773 2381 raft_consensus.cc:740] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.614812 2378 tablet_bootstrap.cc:492] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:44.615061 2381 consensus_queue.cc:260] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.615676 2381 ts_tablet_manager.cc:1434] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.051s user 0.000s sys 0.001s
I20251026 02:56:44.616003 2381 tablet_bootstrap.cc:492] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:44.617426 2381 tablet_bootstrap.cc:654] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.618485 2376 ts_tablet_manager.cc:1929] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.618636 2376 log.cc:1199] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/1d7ffdaeec2240dcbe4cafdb96073a7d
I20251026 02:56:44.619022 2376 ts_tablet_manager.cc:1950] T 1d7ffdaeec2240dcbe4cafdb96073a7d P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:44.619398 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 786) took 4920 ms. Trace:
I20251026 02:56:44.619529 2376 rpcz_store.cc:276] 1026 02:56:39.699046 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.702434 (+ 3388us) service_pool.cc:225] Handling call
1026 02:56:44.439314 (+4736880us) ts_tablet_manager.cc:1151] Deleting tablet 1d7ffdaeec2240dcbe4cafdb96073a7d
1026 02:56:44.439320 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.537345 (+ 98025us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.618363 (+ 81018us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.619378 (+ 1015us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4686554}
I20251026 02:56:44.620141 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 789) took 4919 ms. Trace:
I20251026 02:56:44.620325 2376 rpcz_store.cc:276] 1026 02:56:39.700753 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.704265 (+ 3512us) service_pool.cc:225] Handling call
1026 02:56:44.620002 (+4915737us) ts_tablet_manager.cc:1151] Deleting tablet ec94273925e94fb09f55249426bc4eaa
1026 02:56:44.620011 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.620122 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4867199}
I20251026 02:56:44.620738 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 785) took 4925 ms. Trace:
I20251026 02:56:44.620839 2376 rpcz_store.cc:276] 1026 02:56:39.695373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.705259 (+ 9886us) service_pool.cc:225] Handling call
1026 02:56:44.620607 (+4915348us) ts_tablet_manager.cc:1151] Deleting tablet 7961528688cb41e5be1ba380f954f66a
1026 02:56:44.620618 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.620724 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":117,"tablet-delete.queue_time_us":4867802}
W20251026 02:56:44.621173 2053 consensus_peers.cc:597] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:44.621206 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 791) took 4788 ms. Trace:
I20251026 02:56:44.621279 2376 rpcz_store.cc:276] 1026 02:56:39.832566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.832634 (+ 68us) service_pool.cc:225] Handling call
1026 02:56:44.621096 (+4788462us) ts_tablet_manager.cc:1151] Deleting tablet b1f6e2c58c7f4367813eb66ebda5e0ca
1026 02:56:44.621105 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.621193 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4788254}
I20251026 02:56:44.621516 2376 tablet_replica.cc:333] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:44.621708 2376 raft_consensus.cc:2243] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:44.621831 2378 tablet_bootstrap.cc:654] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.621884 2376 raft_consensus.cc:2272] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.622479 2376 ts_tablet_manager.cc:1916] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.624955 2378 tablet_bootstrap.cc:492] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:44.625972 2378 ts_tablet_manager.cc:1403] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251026 02:56:44.626982 2378 raft_consensus.cc:359] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:44.627339 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88b0cca095514614817346d6eb9014f5 consensus state has no leader
I20251026 02:56:44.628777 2378 raft_consensus.cc:385] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:44.629078 2378 raft_consensus.cc:740] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
W20251026 02:56:44.629596 2195 consensus_peers.cc:597] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7801f073d5d94076af9299492f86fad2. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:44.629732 2378 consensus_queue.cc:260] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.630551 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 1d7ffdaeec2240dcbe4cafdb96073a7d (table default.Table_7b08384413864da0a28224dd7201c263 [id=a9255e57e54144ba876815cbb6155360]) successfully deleted
W20251026 02:56:44.630865 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ec94273925e94fb09f55249426bc4eaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec94273925e94fb09f55249426bc4eaa already in progress: creating tablet
W20251026 02:56:44.631124 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7961528688cb41e5be1ba380f954f66a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7961528688cb41e5be1ba380f954f66a already in progress: creating tablet
W20251026 02:56:44.631364 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b1f6e2c58c7f4367813eb66ebda5e0ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1f6e2c58c7f4367813eb66ebda5e0ca already in progress: creating tablet
I20251026 02:56:44.632994 2378 ts_tablet_manager.cc:1434] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251026 02:56:44.633059 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 6b739a72b2c14b4db6d42b4d63ce224f (DEFAULT_TABLE table=default.Table_251030404f2042a984d8c0dacdf3d560 [id=f1ff26cef73e477baa2ff127ea9bfe35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.634037 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 94fbc54b0dc64488920c5110952a7a81 (DEFAULT_TABLE table=default.Table_35a36ec0f72d4af79182e095d6788ca4 [id=8b4f83c15e9349739b8dd06b8d055620]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.634128 2378 tablet_bootstrap.cc:492] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:44.635457 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 61d46c20e3254369a69dab02b52e12ff (DEFAULT_TABLE table=default.Table_2939f4b1afea4a4b8bbfe83ba4d58558 [id=a955ba1e7317449ba458c8153d3c2343]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.636879 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 163b8f785675474e90ff90a1fe6fc056 (DEFAULT_TABLE table=default.Table_30a03eb811714167a1169b464b284eef [id=3f9214f9eab8482290d619394adbd552]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.637722 3841 raft_consensus.cc:493] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.637723 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a86131ef9a344c92952a42fed41f843e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:44.637923 3841 raft_consensus.cc:515] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:44.638314 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a86131ef9a344c92952a42fed41f843e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:44.638734 2372 ts_tablet_manager.cc:1929] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:44.639223 3841 leader_election.cc:290] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:44.639477 2372 log.cc:1199] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/0cf76ef8f6af445eb0ce15c8d79136c5
I20251026 02:56:44.639423 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e89a98fcfdbb4011876b0ebc2af91f67" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:44.639537 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61d46c20e3254369a69dab02b52e12ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:44.639907 2053 leader_election.cc:343] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.640655 2372 ts_tablet_manager.cc:1950] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:44.640307 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e89a98fcfdbb4011876b0ebc2af91f67" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:44.641115 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b739a72b2c14b4db6d42b4d63ce224f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:44.641600 2195 consensus_peers.cc:597] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 882fe81da21344ed99ea735fa79f73c0. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:44.641162 2053 leader_election.cc:343] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.641747 2053 leader_election.cc:304] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:44.642289 3841 raft_consensus.cc:2749] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:44.642333 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 773) took 6012 ms. Trace:
I20251026 02:56:44.642418 2372 rpcz_store.cc:276] 1026 02:56:38.629788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.630080 (+ 292us) service_pool.cc:225] Handling call
1026 02:56:44.590525 (+5960445us) ts_tablet_manager.cc:1151] Deleting tablet 0cf76ef8f6af445eb0ce15c8d79136c5
1026 02:56:44.590538 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.609038 (+ 18500us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.638626 (+ 29588us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.642315 (+ 3689us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5960226}
I20251026 02:56:44.642800 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 774) took 6003 ms. Trace:
I20251026 02:56:44.642877 2372 rpcz_store.cc:276] 1026 02:56:38.638866 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.638921 (+ 55us) service_pool.cc:225] Handling call
1026 02:56:44.642693 (+6003772us) ts_tablet_manager.cc:1151] Deleting tablet 2f1872ef9f244a388d25a8085dd53f82
1026 02:56:44.642702 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.642784 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6003588}
I20251026 02:56:44.643177 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 775) took 5998 ms. Trace:
I20251026 02:56:44.643255 2372 rpcz_store.cc:276] 1026 02:56:38.644612 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.644667 (+ 55us) service_pool.cc:225] Handling call
1026 02:56:44.643090 (+5998423us) ts_tablet_manager.cc:1151] Deleting tablet 3ac38e6257e74367b06b15fc49d2af81
1026 02:56:44.643099 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.643164 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5998239}
I20251026 02:56:44.643476 2372 tablet_replica.cc:333] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:44.643663 2372 raft_consensus.cc:2243] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:44.643805 2372 raft_consensus.cc:2272] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.644409 2372 ts_tablet_manager.cc:1916] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.646618 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94fbc54b0dc64488920c5110952a7a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.651118 2377 ts_tablet_manager.cc:1929] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251026 02:56:44.652613 3957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:44.652479 2377 log.cc:1199] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/dbf2a1ce78154415b23ab6d0efbc1f52
I20251026 02:56:44.654078 2372 ts_tablet_manager.cc:1929] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:44.654297 2372 log.cc:1199] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/62d30d79a6e4432a8dbe40a104e87fa0
I20251026 02:56:44.654541 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163b8f785675474e90ff90a1fe6fc056. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.654846 2372 ts_tablet_manager.cc:1950] T 62d30d79a6e4432a8dbe40a104e87fa0 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:44.655316 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 776) took 6008 ms. Trace:
I20251026 02:56:44.655480 2372 rpcz_store.cc:276] 1026 02:56:38.646576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.646627 (+ 51us) service_pool.cc:225] Handling call
1026 02:56:44.643439 (+5996812us) ts_tablet_manager.cc:1151] Deleting tablet 62d30d79a6e4432a8dbe40a104e87fa0
1026 02:56:44.643447 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.648937 (+ 5490us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.653940 (+ 5003us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.655300 (+ 1360us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5996651}
I20251026 02:56:44.655961 2372 tablet_replica.cc:333] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:44.656260 2372 raft_consensus.cc:2243] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:44.656533 2372 raft_consensus.cc:2272] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.656695 2377 ts_tablet_manager.cc:1950] T dbf2a1ce78154415b23ab6d0efbc1f52 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:44.657917 2372 ts_tablet_manager.cc:1916] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.659315 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 0cf76ef8f6af445eb0ce15c8d79136c5 (table default.Table_72ee520a0e4d45fb920d1e3cc7a426b3 [id=33334e177bb94b3e9de2a7e7ee59979b]) successfully deleted
W20251026 02:56:44.659610 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 2f1872ef9f244a388d25a8085dd53f82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f1872ef9f244a388d25a8085dd53f82 already in progress: creating tablet
I20251026 02:56:44.660116 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 62d30d79a6e4432a8dbe40a104e87fa0 (table default.Table_5f7b25fab37c4983acf922bd78374da6 [id=777528a5bf944ecb96ebdbea9e7ee559]) successfully deleted
I20251026 02:56:44.660321 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 821) took 5037 ms. Trace:
I20251026 02:56:44.661290 3957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:44.661470 2375 tablet_bootstrap.cc:492] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:44.661634 2375 ts_tablet_manager.cc:1403] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.124s user 0.009s sys 0.011s
I20251026 02:56:44.662392 2375 raft_consensus.cc:359] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.662591 2375 raft_consensus.cc:385] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:44.662675 2375 raft_consensus.cc:740] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.662962 2375 consensus_queue.cc:260] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.661234 2377 rpcz_store.cc:276] 1026 02:56:39.622504 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.624241 (+ 1737us) service_pool.cc:225] Handling call
1026 02:56:44.603533 (+4979292us) ts_tablet_manager.cc:1151] Deleting tablet dbf2a1ce78154415b23ab6d0efbc1f52
1026 02:56:44.603545 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.612470 (+ 8925us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.650303 (+ 37833us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.660035 (+ 9732us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4979108}
I20251026 02:56:44.663723 2375 ts_tablet_manager.cc:1434] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:44.664026 2375 tablet_bootstrap.cc:492] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:44.664842 2377 tablet_replica.cc:333] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:44.665571 2375 tablet_bootstrap.cc:654] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.667265 2377 raft_consensus.cc:2243] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:44.671283 2377 raft_consensus.cc:2272] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.673410 2375 tablet_bootstrap.cc:492] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:44.673632 2375 ts_tablet_manager.cc:1403] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20251026 02:56:44.674319 2375 raft_consensus.cc:359] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.674561 2375 raft_consensus.cc:385] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:44.674659 2375 raft_consensus.cc:740] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.674942 2375 consensus_queue.cc:260] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.675580 2375 ts_tablet_manager.cc:1434] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:44.675892 2375 tablet_bootstrap.cc:492] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:44.677347 2375 tablet_bootstrap.cc:654] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.679247 1663 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_1fa437ef7f89450bb6f672a1fb491baa [id=44c37945ff9f48e8b2c07648394ac177] alter complete (version 1)
I20251026 02:56:44.679551 2377 ts_tablet_manager.cc:1916] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.685138 2375 tablet_bootstrap.cc:492] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:44.685401 2375 ts_tablet_manager.cc:1403] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20251026 02:56:44.686124 2375 raft_consensus.cc:359] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.686367 2375 raft_consensus.cc:385] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:44.686477 2375 raft_consensus.cc:740] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.686769 2375 consensus_queue.cc:260] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.687443 2375 ts_tablet_manager.cc:1434] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:44.687757 2375 tablet_bootstrap.cc:492] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:44.688853 3893 raft_consensus.cc:493] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:44.689002 3893 raft_consensus.cc:515] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.689635 3893 leader_election.cc:290] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:44.690240 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6f9ae327aa4401baf66a737eaa93cf6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:44.690464 2194 consensus_peers.cc:597] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:44.690454 1975 raft_consensus.cc:2468] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:44.690830 2194 leader_election.cc:304] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:44.691057 3893 raft_consensus.cc:2804] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 02:56:44.691139 3893 raft_consensus.cc:493] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:44.691391 3893 raft_consensus.cc:3060] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Advancing to term 2
W20251026 02:56:44.691473 2195 leader_election.cc:343] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: a6f9ae327aa4401baf66a737eaa93cf6
I20251026 02:56:44.692277 2375 tablet_bootstrap.cc:654] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.694131 3893 raft_consensus.cc:515] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.694147 3496 sys_catalog.cc:565] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 02:56:44.695519 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6f9ae327aa4401baf66a737eaa93cf6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:44.695730 1975 raft_consensus.cc:3060] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 2
I20251026 02:56:44.695827 3893 leader_election.cc:290] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:44.698850 1975 raft_consensus.cc:2468] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 2.
I20251026 02:56:44.699335 2194 leader_election.cc:304] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
W20251026 02:56:44.700968 2195 leader_election.cc:343] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: a6f9ae327aa4401baf66a737eaa93cf6
I20251026 02:56:44.701757 3890 raft_consensus.cc:2804] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 02:56:44.701902 3890 raft_consensus.cc:697] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 2 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:44.702132 3890 consensus_queue.cc:237] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.705612 3877 consensus_queue.cc:1048] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:44.718261 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 6b739a72b2c14b4db6d42b4d63ce224f (DEFAULT_TABLE table=default.Table_251030404f2042a984d8c0dacdf3d560 [id=f1ff26cef73e477baa2ff127ea9bfe35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.718973 2372 ts_tablet_manager.cc:1929] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.719748 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 6b739a72b2c14b4db6d42b4d63ce224f (DEFAULT_TABLE table=default.Table_251030404f2042a984d8c0dacdf3d560 [id=f1ff26cef73e477baa2ff127ea9bfe35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.719750 2372 log.cc:1199] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/5c6e3e47b3e7421e945cfbe671b06abe
I20251026 02:56:44.720178 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b739a72b2c14b4db6d42b4d63ce224f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.721155 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 94fbc54b0dc64488920c5110952a7a81 (DEFAULT_TABLE table=default.Table_35a36ec0f72d4af79182e095d6788ca4 [id=8b4f83c15e9349739b8dd06b8d055620]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.722249 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 61d46c20e3254369a69dab02b52e12ff (DEFAULT_TABLE table=default.Table_2939f4b1afea4a4b8bbfe83ba4d58558 [id=a955ba1e7317449ba458c8153d3c2343]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.723423 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 61d46c20e3254369a69dab02b52e12ff (DEFAULT_TABLE table=default.Table_2939f4b1afea4a4b8bbfe83ba4d58558 [id=a955ba1e7317449ba458c8153d3c2343]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.723824 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61d46c20e3254369a69dab02b52e12ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.724133 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 163b8f785675474e90ff90a1fe6fc056 (DEFAULT_TABLE table=default.Table_30a03eb811714167a1169b464b284eef [id=3f9214f9eab8482290d619394adbd552]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.725934 2372 ts_tablet_manager.cc:1950] T 5c6e3e47b3e7421e945cfbe671b06abe P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:44.721792 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 94fbc54b0dc64488920c5110952a7a81 (DEFAULT_TABLE table=default.Table_35a36ec0f72d4af79182e095d6788ca4 [id=8b4f83c15e9349739b8dd06b8d055620]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.728456 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet e45600a92f8e46b096badd700779856d
I20251026 02:56:44.728592 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94fbc54b0dc64488920c5110952a7a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.731957 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet e45600a92f8e46b096badd700779856d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:44.734833 1975 raft_consensus.cc:3060] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:44.737519 2194 consensus_peers.cc:597] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:44.737509 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 777) took 6090 ms. Trace:
I20251026 02:56:44.738966 2372 rpcz_store.cc:276] 1026 02:56:38.647245 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.647289 (+ 44us) service_pool.cc:225] Handling call
1026 02:56:44.655895 (+6008606us) ts_tablet_manager.cc:1151] Deleting tablet 5c6e3e47b3e7421e945cfbe671b06abe
1026 02:56:44.655906 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.684055 (+ 28149us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.718398 (+ 34343us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.737293 (+ 18895us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6008428}
I20251026 02:56:44.742874 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 163b8f785675474e90ff90a1fe6fc056 (DEFAULT_TABLE table=default.Table_30a03eb811714167a1169b464b284eef [id=3f9214f9eab8482290d619394adbd552]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.743795 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet dbf2a1ce78154415b23ab6d0efbc1f52 (table default.Table_22ae79375e8e49d4b6d146d087ffa64b [id=4c170eaba0fd422f9fb0ce5e63e5c746]) successfully deleted
I20251026 02:56:44.744069 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163b8f785675474e90ff90a1fe6fc056. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.744571 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet e45600a92f8e46b096badd700779856d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:44.746243 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3788e42710a0457c80f6ae601866b2ee" } new_table_name: "default.Table_3629bba6ed4d45a292b6c65c325752ec" modify_external_catalogs: true
I20251026 02:56:44.747725 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 5c6e3e47b3e7421e945cfbe671b06abe (table default.Table_653a3f85c24140cdb62a3e9f3ea95fd4 [id=a305da1e3a694b32aff05dbd6c58a6cb]) successfully deleted
I20251026 02:56:44.748376 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet e45600a92f8e46b096badd700779856d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:43 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:44.749702 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 778) took 6098 ms. Trace:
I20251026 02:56:44.751827 3877 consensus_queue.cc:1048] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:44.751070 2372 rpcz_store.cc:276] 1026 02:56:38.651368 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.651433 (+ 65us) service_pool.cc:225] Handling call
1026 02:56:44.746663 (+6095230us) ts_tablet_manager.cc:1151] Deleting tablet 863aef239cd34a358a71ddfdba6fb72a
1026 02:56:44.746933 (+ 270us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.749491 (+ 2558us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6094778}
I20251026 02:56:44.753538 1662 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_c4061134bbfd4244a23d6c37781dcb86 [id=5333d696902241b89eadf7e9e627c41a] alter complete (version 1)
I20251026 02:56:44.753911 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet 7e1a45412db1449a99252112d8110a17 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:44.755483 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 863aef239cd34a358a71ddfdba6fb72a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 863aef239cd34a358a71ddfdba6fb72a already in progress: creating tablet
I20251026 02:56:44.756649 2372 tablet_replica.cc:333] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:44.759135 2372 raft_consensus.cc:2243] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:44.761071 2377 ts_tablet_manager.cc:1929] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.762426 2372 raft_consensus.cc:2272] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.763554 2377 log.cc:1199] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/d65b0d79e2e54af9810022b683c3288b
I20251026 02:56:44.769095 2377 ts_tablet_manager.cc:1950] T d65b0d79e2e54af9810022b683c3288b P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:44.773620 2372 ts_tablet_manager.cc:1916] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.774128 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 822) took 5149 ms. Trace:
I20251026 02:56:44.775556 2377 rpcz_store.cc:276] 1026 02:56:39.624554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.624589 (+ 35us) service_pool.cc:225] Handling call
1026 02:56:44.664536 (+5039947us) ts_tablet_manager.cc:1151] Deleting tablet d65b0d79e2e54af9810022b683c3288b
1026 02:56:44.664551 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.728969 (+ 64418us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.760131 (+ 31162us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.773790 (+ 13659us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":195,"tablet-delete.queue_time_us":5039565}
I20251026 02:56:44.779994 2381 tablet_bootstrap.cc:492] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:44.780663 2381 ts_tablet_manager.cc:1403] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.165s user 0.003s sys 0.000s
I20251026 02:56:44.781728 2381 raft_consensus.cc:359] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.782018 2381 raft_consensus.cc:385] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:44.782151 2377 tablet_replica.cc:333] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:44.784173 2377 raft_consensus.cc:2243] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 2 LEADER]: Raft consensus shutting down.
I20251026 02:56:44.786306 2377 pending_rounds.cc:70] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:44.786738 2377 pending_rounds.cc:77] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7214888568073482240 op_type: NO_OP noop_request { }
I20251026 02:56:44.787515 2377 raft_consensus.cc:2889] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:44.788342 2377 raft_consensus.cc:2272] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.789556 1980 raft_consensus.cc:3060] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.789734 2082 tablet_service.cc:1505] Processing CreateTablet for tablet b2eab97dc2b9409697589a34e9cc7d6c (DEFAULT_TABLE table=default.Table_1eca935882b04878bb1362a022e82ecf [id=732d51d0d5174293985c760d55f6f46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.790134 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2eab97dc2b9409697589a34e9cc7d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.793337 2377 ts_tablet_manager.cc:1916] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.805931 1662 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 41a1442fffe54add93666e3c78a0cc58 (table default.Table_1faa0c8151724c5e816546ee2e1d5182 [id=a106f9ece0b9467a8bc1d5d4b3524c41]). Expected version 1 got 0
I20251026 02:56:44.806182 1662 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 398c1f420974453b9b6b0fe17795378b (table default.Table_a92e68d05f4b455d810180ee4cc5874d [id=7f64fbec3582415a8924d01e14856a56]). Expected version 1 got 0
W20251026 02:56:44.813987 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 94fbc54b0dc64488920c5110952a7a81 consensus state has no leader
I20251026 02:56:44.816040 3841 raft_consensus.cc:493] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.816246 3841 raft_consensus.cc:515] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.816978 3841 leader_election.cc:290] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:44.817593 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e203fe0403594d30b44d531153d42a10" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:44.817849 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e203fe0403594d30b44d531153d42a10" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:44.818507 2053 leader_election.cc:343] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:44.818768 2053 leader_election.cc:343] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.818856 2053 leader_election.cc:304] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:44.819413 3841 raft_consensus.cc:2749] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:44.825843 3372 raft_consensus.cc:493] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.826032 3372 raft_consensus.cc:515] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.826749 3372 leader_election.cc:290] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:44.828007 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfa68f2c942c43e2a02e9d333bdfa2a2" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:44.828884 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfa68f2c942c43e2a02e9d333bdfa2a2" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:44.829636 1916 leader_election.cc:304] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:44.830103 3372 raft_consensus.cc:2749] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:44.830770 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94fbc54b0dc64488920c5110952a7a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.832350 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b739a72b2c14b4db6d42b4d63ce224f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.832844 2378 tablet_bootstrap.cc:654] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.833935 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61d46c20e3254369a69dab02b52e12ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.834460 2240 tablet_service.cc:1505] Processing CreateTablet for tablet b2eab97dc2b9409697589a34e9cc7d6c (DEFAULT_TABLE table=default.Table_1eca935882b04878bb1362a022e82ecf [id=732d51d0d5174293985c760d55f6f46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.834879 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2eab97dc2b9409697589a34e9cc7d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.836344 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163b8f785675474e90ff90a1fe6fc056. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.840441 1948 tablet_service.cc:1505] Processing CreateTablet for tablet b2eab97dc2b9409697589a34e9cc7d6c (DEFAULT_TABLE table=default.Table_1eca935882b04878bb1362a022e82ecf [id=732d51d0d5174293985c760d55f6f46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.840891 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2eab97dc2b9409697589a34e9cc7d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.841022 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2b3b66c0fb3a4f10bf684d40855009d7" } modify_external_catalogs: true
I20251026 02:56:44.841253 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2b3b66c0fb3a4f10bf684d40855009d7" } modify_external_catalogs: true
W20251026 02:56:44.848265 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7b5619d7b7243a99aff0c5fb29e499e consensus state has no leader
I20251026 02:56:44.854697 2082 tablet_service.cc:1505] Processing CreateTablet for tablet a43a38d09625499197aefbe7c9ac3af4 (DEFAULT_TABLE table=default.Table_64502b8f2393429f85256d6cbdaf7148 [id=e79b86ea67774d4aa63294d3b6cfaa40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.860888 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet d65b0d79e2e54af9810022b683c3288b (table default.Table_bcac335a2c9a479fb5dea86b2aeffde6 [id=30a5783033cb4f7f929e766fb47d92ff]) successfully deleted
I20251026 02:56:44.862270 1948 tablet_service.cc:1505] Processing CreateTablet for tablet a43a38d09625499197aefbe7c9ac3af4 (DEFAULT_TABLE table=default.Table_64502b8f2393429f85256d6cbdaf7148 [id=e79b86ea67774d4aa63294d3b6cfaa40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.863500 2242 tablet_service.cc:1505] Processing CreateTablet for tablet a43a38d09625499197aefbe7c9ac3af4 (DEFAULT_TABLE table=default.Table_64502b8f2393429f85256d6cbdaf7148 [id=e79b86ea67774d4aa63294d3b6cfaa40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.863955 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43a38d09625499197aefbe7c9ac3af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.864245 2378 tablet_bootstrap.cc:492] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:44.864477 2378 ts_tablet_manager.cc:1403] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.230s user 0.010s sys 0.026s
I20251026 02:56:44.782166 2381 raft_consensus.cc:740] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.865355 2378 raft_consensus.cc:359] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.865653 2378 raft_consensus.cc:385] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:44.865614 2381 consensus_queue.cc:260] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.865754 2378 raft_consensus.cc:740] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.866050 2378 consensus_queue.cc:260] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.866523 2381 ts_tablet_manager.cc:1434] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.086s user 0.001s sys 0.001s
I20251026 02:56:44.866621 2378 ts_tablet_manager.cc:1434] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:44.867511 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43a38d09625499197aefbe7c9ac3af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.871053 3888 raft_consensus.cc:493] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:44.871403 3888 raft_consensus.cc:515] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.872601 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a012d868a78f4324a92db83c81931b9d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:44.873204 2194 leader_election.cc:343] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.873593 3888 leader_election.cc:290] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:44.873786 2378 tablet_bootstrap.cc:492] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:44.875625 2378 tablet_bootstrap.cc:654] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.875690 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a012d868a78f4324a92db83c81931b9d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:44.876003 2116 raft_consensus.cc:2468] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:44.876452 2195 leader_election.cc:304] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:44.877314 3893 raft_consensus.cc:2804] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:44.877471 3893 raft_consensus.cc:493] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:44.877604 3893 raft_consensus.cc:3060] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.877552 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a27cf0a8326c4683a305ae448369f2cd" } modify_external_catalogs: true
I20251026 02:56:44.877756 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a27cf0a8326c4683a305ae448369f2cd" } modify_external_catalogs: true
I20251026 02:56:44.881553 3893 raft_consensus.cc:515] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.883431 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a012d868a78f4324a92db83c81931b9d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:44.883677 2116 raft_consensus.cc:3060] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.882675 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a012d868a78f4324a92db83c81931b9d" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:44.886276 2194 leader_election.cc:343] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:44.886570 2116 raft_consensus.cc:2468] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:44.887038 2195 leader_election.cc:304] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:44.887441 3890 raft_consensus.cc:2804] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:44.887919 3890 raft_consensus.cc:697] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:44.888209 3890 consensus_queue.cc:237] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.890089 3893 leader_election.cc:290] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:44.893438 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43a38d09625499197aefbe7c9ac3af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.894704 2372 ts_tablet_manager.cc:1929] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.894834 2372 log.cc:1199] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7177931353164b2aa82c00b2807ccd63
I20251026 02:56:44.900283 2308 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:44.900661 2381 tablet_bootstrap.cc:492] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:44.901989 2372 ts_tablet_manager.cc:1950] T 7177931353164b2aa82c00b2807ccd63 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:44.903254 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 147cb1f5d1a4477fb253f17d252e59e7 (DEFAULT_TABLE table=default.Table_925beaee6cd548618705f64874644a54 [id=ed68febe7f6d447e9b84ede641521037]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.907516 2381 tablet_bootstrap.cc:654] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:44.910522 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 2f4982d068e946808b5820288aefc212 (DEFAULT_TABLE table=default.Table_b703a0b78fda4c0597aded8f5e8ccf6f [id=462d2abfb0b64dbca5eb168caa0edc2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.912426 2094 tablet_service.cc:1505] Processing CreateTablet for tablet a0dd76717299489294d53f3fe2ca6eb5 (DEFAULT_TABLE table=default.Table_bff4472649e4461a8f829b44bb15da12 [id=eefa097a8ae7437db37468e476a1b4c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.914049 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 06b5ac1b5e744170b4b455f3d7af092d (DEFAULT_TABLE table=default.Table_fb320a7039df4723a13ea9706037a238 [id=f3a7597159fb4e798920bcf0d285c98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.914639 1975 raft_consensus.cc:3060] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:44.916671 2377 ts_tablet_manager.cc:1929] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251026 02:56:44.916806 2377 log.cc:1199] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/ae95654d129441e08fcb3f3b355ffc22
I20251026 02:56:44.917169 2377 ts_tablet_manager.cc:1950] T ae95654d129441e08fcb3f3b355ffc22 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:44.917775 2375 tablet_bootstrap.cc:492] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:44.924432 2375 ts_tablet_manager.cc:1403] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.237s user 0.008s sys 0.004s
I20251026 02:56:44.924417 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_555ef24f609a48feaeb346a1523c97d0" } modify_external_catalogs: true
I20251026 02:56:44.924649 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_555ef24f609a48feaeb346a1523c97d0" } modify_external_catalogs: true
I20251026 02:56:44.928349 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 827) took 5300 ms. Trace:
I20251026 02:56:44.928465 2377 rpcz_store.cc:276] 1026 02:56:39.627534 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.627884 (+ 350us) service_pool.cc:225] Handling call
1026 02:56:44.781823 (+5153939us) ts_tablet_manager.cc:1151] Deleting tablet ae95654d129441e08fcb3f3b355ffc22
1026 02:56:44.781849 (+ 26us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.825583 (+ 43734us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.916552 (+ 90969us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.928320 (+ 11768us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5153528}
I20251026 02:56:44.929917 2375 raft_consensus.cc:359] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.934295 2375 raft_consensus.cc:385] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:44.934482 2375 raft_consensus.cc:740] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:44.934530 2377 tablet_replica.cc:333] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:44.934729 2377 raft_consensus.cc:2243] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:44.934901 2375 consensus_queue.cc:260] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:44.935079 2377 raft_consensus.cc:2272] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:44.935662 2377 ts_tablet_manager.cc:1916] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:44.935792 2375 ts_tablet_manager.cc:1434] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251026 02:56:44.938534 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 147cb1f5d1a4477fb253f17d252e59e7 (DEFAULT_TABLE table=default.Table_925beaee6cd548618705f64874644a54 [id=ed68febe7f6d447e9b84ede641521037]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.938683 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 147cb1f5d1a4477fb253f17d252e59e7 (DEFAULT_TABLE table=default.Table_925beaee6cd548618705f64874644a54 [id=ed68febe7f6d447e9b84ede641521037]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.939023 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 147cb1f5d1a4477fb253f17d252e59e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.939301 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 2f4982d068e946808b5820288aefc212 (DEFAULT_TABLE table=default.Table_b703a0b78fda4c0597aded8f5e8ccf6f [id=462d2abfb0b64dbca5eb168caa0edc2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.939523 1962 tablet_service.cc:1505] Processing CreateTablet for tablet a0dd76717299489294d53f3fe2ca6eb5 (DEFAULT_TABLE table=default.Table_bff4472649e4461a8f829b44bb15da12 [id=eefa097a8ae7437db37468e476a1b4c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.939742 2232 tablet_service.cc:1505] Processing CreateTablet for tablet a0dd76717299489294d53f3fe2ca6eb5 (DEFAULT_TABLE table=default.Table_bff4472649e4461a8f829b44bb15da12 [id=eefa097a8ae7437db37468e476a1b4c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.939862 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0dd76717299489294d53f3fe2ca6eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.940624 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 06b5ac1b5e744170b4b455f3d7af092d (DEFAULT_TABLE table=default.Table_fb320a7039df4723a13ea9706037a238 [id=f3a7597159fb4e798920bcf0d285c98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.940764 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet ae95654d129441e08fcb3f3b355ffc22 (table default.Table_8602d75889ea4576ae575b55fe95e940 [id=b0104213af32435589ae19109b7aee7f]) successfully deleted
I20251026 02:56:44.941051 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 06b5ac1b5e744170b4b455f3d7af092d (DEFAULT_TABLE table=default.Table_fb320a7039df4723a13ea9706037a238 [id=f3a7597159fb4e798920bcf0d285c98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.941426 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b5ac1b5e744170b4b455f3d7af092d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.944495 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 2f4982d068e946808b5820288aefc212 (DEFAULT_TABLE table=default.Table_b703a0b78fda4c0597aded8f5e8ccf6f [id=462d2abfb0b64dbca5eb168caa0edc2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:44.944872 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4982d068e946808b5820288aefc212. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:44.946424 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3ee83bc1426c40dfb0c28e58715a9476" } modify_external_catalogs: true
I20251026 02:56:44.946655 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3ee83bc1426c40dfb0c28e58715a9476" } modify_external_catalogs: true
W20251026 02:56:44.955613 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3131bc2948e34c25a1bbda90f8dabeb3 consensus state has no leader
I20251026 02:56:44.957561 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4e364f533870440b9c9390bb6091228e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:44.958205 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e364f533870440b9c9390bb6091228e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:44.963514 2377 ts_tablet_manager.cc:1929] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:44.972373 2377 log.cc:1199] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3b7d85a2941545ceaa16b22e8328b4ff
I20251026 02:56:44.972782 2377 ts_tablet_manager.cc:1950] T 3b7d85a2941545ceaa16b22e8328b4ff P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:44.975430 2378 tablet_bootstrap.cc:492] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:44.975664 2378 ts_tablet_manager.cc:1403] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.102s user 0.008s sys 0.007s
I20251026 02:56:44.981874 3372 raft_consensus.cc:493] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 696476265495492790739ec126a08539)
I20251026 02:56:44.982074 3372 raft_consensus.cc:515] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:44.982731 3372 leader_election.cc:290] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 696476265495492790739ec126a08539 (127.1.150.3:38107), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:44.986508 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_804ab36efead4908af699d1174155449"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:44.987197 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_804ab36efead4908af699d1174155449 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:44.993988 1663 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_0d155486332941679014bf465dbbb2de [id=e305b9beb6b74a33b3b7c64f34be8cb9] alter complete (version 1)
W20251026 02:56:44.994731 1663 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.3:36919 (request call id 126) took 879 ms (client timeout 1000 ms). Trace:
W20251026 02:56:44.994880 1663 rpcz_store.cc:269] 1026 02:56:44.115663 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.115720 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:44.994694 (+878974us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:44.673382 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:44.673528 (+ 146us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:44.673534 (+ 6us) write_op.cc:435] Acquired schema lock
1026 02:56:44.673537 (+ 3us) tablet.cc:660] Decoding operations
1026 02:56:44.673660 (+ 123us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:44.673675 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:44.673680 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1026 02:56:44.673751 (+ 71us) tablet.cc:699] Row locks acquired
1026 02:56:44.673754 (+ 3us) write_op.cc:260] PREPARE: finished
1026 02:56:44.673799 (+ 45us) write_op.cc:270] Start()
1026 02:56:44.673837 (+ 38us) write_op.cc:276] Timestamp: P: 1761447404673723 usec, L: 0
1026 02:56:44.673841 (+ 4us) op_driver.cc:348] REPLICATION: starting
1026 02:56:44.674303 (+ 462us) log.cc:844] Serialized 356 byte log entry
1026 02:56:44.678461 (+ 4158us) op_driver.cc:464] REPLICATION: finished
1026 02:56:44.678649 (+ 188us) write_op.cc:301] APPLY: starting
1026 02:56:44.678666 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:44.678712 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:44.678739 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:44.678796 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:44.678836 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:44.678844 (+ 8us) write_op.cc:312] APPLY: finished
1026 02:56:44.678930 (+ 86us) log.cc:844] Serialized 23 byte log entry
1026 02:56:44.678996 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:44.679018 (+ 22us) write_op.cc:454] Released schema lock
1026 02:56:44.679027 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:44.716163 (+ 37136us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:44.716175 (+ 12us) write_op.cc:454] Released schema lock
Related trace 'op':
1026 02:56:44.988493 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:44.988623 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:44.988628 (+ 5us) write_op.cc:435] Acquired schema lock
1026 02:56:44.988630 (+ 2us) tablet.cc:660] Decoding operations
1026 02:56:44.988770 (+ 140us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:44.988797 (+ 27us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:44.988801 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1026 02:56:44.988866 (+ 65us) tablet.cc:699] Row locks acquired
1026 02:56:44.988868 (+ 2us) write_op.cc:260] PREPARE: finished
1026 02:56:44.988902 (+ 34us) write_op.cc:270] Start()
1026 02:56:44.988938 (+ 36us) write_op.cc:276] Timestamp: P: 1761447404988876 usec, L: 0
1026 02:56:44.988942 (+ 4us) op_driver.cc:348] REPLICATION: starting
1026 02:56:44.989161 (+ 219us) log.cc:844] Serialized 356 byte log entry
1026 02:56:44.993237 (+ 4076us) op_driver.cc:464] REPLICATION: finished
1026 02:56:44.993426 (+ 189us) write_op.cc:301] APPLY: starting
1026 02:56:44.993443 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:44.993487 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:44.993512 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:44.993569 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:44.993602 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:44.993610 (+ 8us) write_op.cc:312] APPLY: finished
1026 02:56:44.993686 (+ 76us) log.cc:844] Serialized 23 byte log entry
1026 02:56:44.993744 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:44.993765 (+ 21us) write_op.cc:454] Released schema lock
1026 02:56:44.993772 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":135,"apply.run_cpu_time_us":445,"apply.run_wall_time_us":444,"num_ops":1,"prepare.queue_time_us":459260,"prepare.run_cpu_time_us":872,"prepare.run_wall_time_us":42755,"raft.queue_time_us":43,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":247,"replication_time_us":4586}],["op",{"apply.queue_time_us":132,"apply.run_cpu_time_us":413,"apply.run_wall_time_us":413,"num_ops":1,"prepare.queue_time_us":307762,"raft.queue_time_us":35,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":247,"replication_time_us":4265}]]}
I20251026 02:56:45.002110 3372 raft_consensus.cc:493] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.002285 3372 raft_consensus.cc:515] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.002959 3372 leader_election.cc:290] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.003701 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:45.003959 2116 raft_consensus.cc:2393] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 0b6b0a6c4e144982b67980118cddba48 in this term.
W20251026 02:56:45.005143 1916 leader_election.cc:343] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 6108219254bd448c852096c0c967384b
I20251026 02:56:45.006310 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet d90e62ae86b149b291dc29cf7442c6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:45.006542 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4982d068e946808b5820288aefc212. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.006857 2375 tablet_bootstrap.cc:492] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:45.008435 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0dd76717299489294d53f3fe2ca6eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.008459 2375 tablet_bootstrap.cc:654] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.009796 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c6e3e47b3e7421e945cfbe671b06abe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:45.010366 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 7177931353164b2aa82c00b2807ccd63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.011402 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet ed836bb7519e4b0290e6547fa57128bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.011876 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:45.012106 2259 raft_consensus.cc:2393] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:45.012431 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b5ac1b5e744170b4b455f3d7af092d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:45.012683 1915 leader_election.cc:343] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 6108219254bd448c852096c0c967384b
I20251026 02:56:45.012799 1915 leader_election.cc:304] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:45.013223 1915 leader_election.cc:304] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:45.016445 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 147cb1f5d1a4477fb253f17d252e59e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.016991 3372 raft_consensus.cc:2749] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:45.016996 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 779) took 6353 ms. Trace:
I20251026 02:56:45.017179 3372 raft_consensus.cc:3060] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:45.017103 2372 rpcz_store.cc:276] 1026 02:56:38.663342 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.663399 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:44.755820 (+6092421us) ts_tablet_manager.cc:1151] Deleting tablet 7177931353164b2aa82c00b2807ccd63
1026 02:56:44.756013 (+ 193us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.877333 (+121320us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.894594 (+ 17261us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.016972 (+122378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6092035}
I20251026 02:56:45.017586 2372 tablet_replica.cc:333] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:45.017855 2372 raft_consensus.cc:2243] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:45.018082 2372 raft_consensus.cc:2272] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.018987 2372 ts_tablet_manager.cc:1916] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.020582 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7177931353164b2aa82c00b2807ccd63 (table default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 [id=7de20c0869584b56b936144d62202444]) successfully deleted
I20251026 02:56:45.022920 3372 raft_consensus.cc:2749] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:45.027302 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 147cb1f5d1a4477fb253f17d252e59e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.031801 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4982d068e946808b5820288aefc212. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.031621 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ef4b978686a546d896741fa10badbddd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:45.032280 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef4b978686a546d896741fa10badbddd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:45.033738 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_98144cf085204d6ba89b8612ab94c4b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:45.034258 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98144cf085204d6ba89b8612ab94c4b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:45.036676 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0dd76717299489294d53f3fe2ca6eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.038620 2378 raft_consensus.cc:359] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.039608 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 828) took 5412 ms. Trace:
I20251026 02:56:45.039710 2377 rpcz_store.cc:276] 1026 02:56:39.627613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.628076 (+ 463us) service_pool.cc:225] Handling call
1026 02:56:44.931028 (+5302952us) ts_tablet_manager.cc:1151] Deleting tablet 3b7d85a2941545ceaa16b22e8328b4ff
1026 02:56:44.934470 (+ 3442us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.938742 (+ 4272us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:44.963397 (+ 24655us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.039586 (+ 76189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5302825}
I20251026 02:56:45.039865 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b5ac1b5e744170b4b455f3d7af092d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.040117 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 832) took 5342 ms. Trace:
I20251026 02:56:45.040232 2377 rpcz_store.cc:276] 1026 02:56:39.697960 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.698046 (+ 86us) service_pool.cc:225] Handling call
1026 02:56:45.039988 (+5341942us) ts_tablet_manager.cc:1151] Deleting tablet 7961528688cb41e5be1ba380f954f66a
1026 02:56:45.039997 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.040100 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5341753}
I20251026 02:56:45.040616 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 836) took 5339 ms. Trace:
I20251026 02:56:45.040709 2377 rpcz_store.cc:276] 1026 02:56:39.700827 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.704405 (+ 3578us) service_pool.cc:225] Handling call
1026 02:56:45.040511 (+5336106us) ts_tablet_manager.cc:1151] Deleting tablet 144b957c69e64085a6f78050dea426b4
1026 02:56:45.040522 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.040596 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5335924}
I20251026 02:56:45.041018 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 837) took 5339 ms. Trace:
I20251026 02:56:45.041083 2377 rpcz_store.cc:276] 1026 02:56:39.701829 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.704742 (+ 2913us) service_pool.cc:225] Handling call
1026 02:56:45.040911 (+5336169us) ts_tablet_manager.cc:1151] Deleting tablet ec94273925e94fb09f55249426bc4eaa
1026 02:56:45.040923 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.041003 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5336019}
I20251026 02:56:45.041373 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 831) took 5346 ms. Trace:
I20251026 02:56:45.041455 2377 rpcz_store.cc:276] 1026 02:56:39.694955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.704886 (+ 9931us) service_pool.cc:225] Handling call
1026 02:56:45.041275 (+5336389us) ts_tablet_manager.cc:1151] Deleting tablet c2d6691bb4cf4a8292b0c082e54a91a5
1026 02:56:45.041285 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.041357 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5336261}
I20251026 02:56:45.041733 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 830) took 5348 ms. Trace:
I20251026 02:56:45.041810 2377 rpcz_store.cc:276] 1026 02:56:39.692964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.705406 (+ 12442us) service_pool.cc:225] Handling call
1026 02:56:45.041640 (+5336234us) ts_tablet_manager.cc:1151] Deleting tablet 112181b9207e448f959f8311ad95a02b
1026 02:56:45.041647 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.041716 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5336095}
I20251026 02:56:45.042112 2377 tablet_replica.cc:333] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:45.042332 2377 raft_consensus.cc:2243] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:45.042685 2377 raft_consensus.cc:2272] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.043262 2377 ts_tablet_manager.cc:1916] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:45.046540 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7bef60e3a10843b4be2022ab4b64be68 consensus state has no leader
I20251026 02:56:45.048591 2082 tablet_service.cc:1505] Processing CreateTablet for tablet e3d58b8297604edea20f9eb16cc9b6bf (DEFAULT_TABLE table=default.Table_dab46018e72e4863ba644a72c0fcdbf9 [id=87b37d02a6b84e618358af7db1db75fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.054113 2377 ts_tablet_manager.cc:1929] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.054409 2377 log.cc:1199] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7801f073d5d94076af9299492f86fad2
I20251026 02:56:45.055112 2377 ts_tablet_manager.cc:1950] T 7801f073d5d94076af9299492f86fad2 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:45.056015 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 834) took 5356 ms. Trace:
I20251026 02:56:45.055876 1654 catalog_manager.cc:5649] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:45.056387 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (table default.Table_30611e161c274e8b99bd1251e5e527aa [id=b5d91d3e8a2643c09fee85976972391a]). Expected version 1 got 0
I20251026 02:56:45.056938 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet e9cd10092aac40068a1c7f85e514fd37 (table default.Table_fe09be2527a840b09e3e64c27965845d [id=de5272bcc29644208ddd95bdec21254e]). Expected version 1 got 0
I20251026 02:56:45.057277 1654 catalog_manager.cc:5649] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:45.060037 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 23062e7292f44ad5ad9dfc22a401741c (table default.Table_11cf708e0e8243c887777e42acc25490 [id=2e17bae10e0b48a492f5ee0bed7d00d0]). Expected version 1 got 0
I20251026 02:56:45.064052 2378 raft_consensus.cc:385] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.064251 2378 raft_consensus.cc:740] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.064640 2378 consensus_queue.cc:260] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.065320 2378 ts_tablet_manager.cc:1434] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.090s user 0.002s sys 0.000s
I20251026 02:56:45.065656 2378 tablet_bootstrap.cc:492] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:45.061614 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 48401.7us
I20251026 02:56:45.040001 1945 tablet_service.cc:1505] Processing CreateTablet for tablet e3d58b8297604edea20f9eb16cc9b6bf (DEFAULT_TABLE table=default.Table_dab46018e72e4863ba644a72c0fcdbf9 [id=87b37d02a6b84e618358af7db1db75fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.066627 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3d58b8297604edea20f9eb16cc9b6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.069962 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 43f10bd71013475baf8c62bbb3885bd9 (DEFAULT_TABLE table=default.Table_44584371880e4feeb01066c835d98432 [id=d01e3f7e40ce483f90d60dfb78a19028]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.070406 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43f10bd71013475baf8c62bbb3885bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.070938 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3b7d85a2941545ceaa16b22e8328b4ff (table default.Table_5ae1979aa25849fe8b780ecc617bb7e8 [id=9f6ac6400247491485d6bef30bf6cf4b]) successfully deleted
I20251026 02:56:45.071167 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 5f29197346704f84ba6e3dcb2f5b8bd4 (DEFAULT_TABLE table=default.Table_622b499e485f4f00945466ab4023cc0e [id=66564f3557d349238a2d9701af21c80b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:45.071272 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7961528688cb41e5be1ba380f954f66a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7961528688cb41e5be1ba380f954f66a already in progress: creating tablet
I20251026 02:56:45.071534 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f29197346704f84ba6e3dcb2f5b8bd4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:45.071714 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 144b957c69e64085a6f78050dea426b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 144b957c69e64085a6f78050dea426b4 already in progress: creating tablet
I20251026 02:56:45.087853 3841 raft_consensus.cc:493] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.088040 3841 raft_consensus.cc:515] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.063858 2229 tablet_service.cc:1505] Processing CreateTablet for tablet e3d58b8297604edea20f9eb16cc9b6bf (DEFAULT_TABLE table=default.Table_dab46018e72e4863ba644a72c0fcdbf9 [id=87b37d02a6b84e618358af7db1db75fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.070073 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 43f10bd71013475baf8c62bbb3885bd9 (DEFAULT_TABLE table=default.Table_44584371880e4feeb01066c835d98432 [id=d01e3f7e40ce483f90d60dfb78a19028]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.088245 2377 rpcz_store.cc:276] 1026 02:56:39.699994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.704340 (+ 4346us) service_pool.cc:225] Handling call
1026 02:56:45.042070 (+5337730us) ts_tablet_manager.cc:1151] Deleting tablet 7801f073d5d94076af9299492f86fad2
1026 02:56:45.042080 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.046684 (+ 4604us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.053986 (+ 7302us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.055994 (+ 2008us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5306464}
I20251026 02:56:45.088577 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43f10bd71013475baf8c62bbb3885bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.088754 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 839) took 5255 ms. Trace:
I20251026 02:56:45.088771 3841 leader_election.cc:290] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.088827 2377 rpcz_store.cc:276] 1026 02:56:39.833146 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.833423 (+ 277us) service_pool.cc:225] Handling call
1026 02:56:45.088627 (+5255204us) ts_tablet_manager.cc:1151] Deleting tablet b1f6e2c58c7f4367813eb66ebda5e0ca
1026 02:56:45.088636 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.088738 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5208253}
I20251026 02:56:45.089077 2377 tablet_replica.cc:333] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:45.089263 2377 raft_consensus.cc:2243] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:45.089622 2377 raft_consensus.cc:2272] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.089968 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "398c1f420974453b9b6b0fe17795378b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:45.090178 2377 ts_tablet_manager.cc:1916] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:45.090654 2053 leader_election.cc:343] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.091377 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "398c1f420974453b9b6b0fe17795378b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:45.091913 2053 leader_election.cc:343] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.092031 2053 leader_election.cc:304] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:45.092553 3841 raft_consensus.cc:2749] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:45.070851 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 5f29197346704f84ba6e3dcb2f5b8bd4 (DEFAULT_TABLE table=default.Table_622b499e485f4f00945466ab4023cc0e [id=66564f3557d349238a2d9701af21c80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.094787 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f29197346704f84ba6e3dcb2f5b8bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.101851 2381 tablet_bootstrap.cc:492] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:45.102036 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 43f10bd71013475baf8c62bbb3885bd9 (DEFAULT_TABLE table=default.Table_44584371880e4feeb01066c835d98432 [id=d01e3f7e40ce483f90d60dfb78a19028]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.102118 2381 ts_tablet_manager.cc:1403] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.202s user 0.015s sys 0.023s
I20251026 02:56:45.102520 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3d58b8297604edea20f9eb16cc9b6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.102527 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 5f29197346704f84ba6e3dcb2f5b8bd4 (DEFAULT_TABLE table=default.Table_622b499e485f4f00945466ab4023cc0e [id=66564f3557d349238a2d9701af21c80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.102849 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 2d75ea3e7a6c4aadb438f9e24d9d6a48 (DEFAULT_TABLE table=default.Table_00a6c982df6b4e6892bc060e529d5ee4 [id=9ce8a1e081c04d8d98c4e90f0e467e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.104405 1980 raft_consensus.cc:3060] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:45.116612 2372 ts_tablet_manager.cc:1929] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.116786 2372 log.cc:1199] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/f437d8b5f5784e8fbec10d0700e90348
I20251026 02:56:45.117219 2372 ts_tablet_manager.cc:1950] T f437d8b5f5784e8fbec10d0700e90348 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:45.119010 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3d58b8297604edea20f9eb16cc9b6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.127933 2376 ts_tablet_manager.cc:1929] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.132148 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 3865399cf6934bc18e16ae2cb4887197 (DEFAULT_TABLE table=default.Table_2b3b66c0fb3a4f10bf684d40855009d7 [id=bd7ee261e0154f0195abc99ca9790a28]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:45.133857 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ec94273925e94fb09f55249426bc4eaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec94273925e94fb09f55249426bc4eaa already in progress: creating tablet
W20251026 02:56:45.134032 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 62069.1us
W20251026 02:56:45.134289 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet c2d6691bb4cf4a8292b0c082e54a91a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2d6691bb4cf4a8292b0c082e54a91a5 already in progress: creating tablet
W20251026 02:56:45.134476 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 112181b9207e448f959f8311ad95a02b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 112181b9207e448f959f8311ad95a02b already in progress: creating tablet
I20251026 02:56:45.134968 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7801f073d5d94076af9299492f86fad2 (table default.Table_03a9f618938a45aab881c91a801cc32c [id=12b41b47b5d243188eb73bd83a83bbdc]) successfully deleted
I20251026 02:56:45.135159 2095 tablet_service.cc:1505] Processing CreateTablet for tablet a02c9d84f0ec498e896ec97a75e3c154 (DEFAULT_TABLE table=default.Table_2d20995352a745b1a63b0cb745126074 [id=2f89068c1a7b4989b4e5faa8e1cc19c0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:45.135293 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet b1f6e2c58c7f4367813eb66ebda5e0ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1f6e2c58c7f4367813eb66ebda5e0ca already in progress: creating tablet
I20251026 02:56:45.135619 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 2e26156d5a154c08afbd8b1e25a4dfe7 (DEFAULT_TABLE table=default.Table_9b0e2338f40441e6b644b1e7740df97b [id=e47c89a8209646f4bb0e8cd5057a8a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.137094 2100 tablet_service.cc:1505] Processing CreateTablet for tablet f27da2b726344153b54c0187660ca39f (DEFAULT_TABLE table=default.Table_016a71d765444bc78bf9e3cf36571b0b [id=cd1317673c124e7a9a9643b45b2ebd3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.137161 2098 tablet_service.cc:1505] Processing CreateTablet for tablet bb1ec261247744c3bde2ea429e13b19d (DEFAULT_TABLE table=default.Table_9dc1a989050c4cf48b9a8111a722b2ee [id=ddcd017fd0dd4fbdbacdcd370b9085c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.138037 2089 tablet_service.cc:1505] Processing CreateTablet for tablet 917851a613af4ae3a6e09b4be14447e0 (DEFAULT_TABLE table=default.Table_5b06ba4234b24bf59077dcc622039de2 [id=2abf870df25741b49e289e3ec9d9d474]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.138370 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 93d8db69c285430687f5e99110f4bfad (DEFAULT_TABLE table=default.Table_3478623ef36e4ef7910a71260ba0ecb4 [id=bb1263d8e28f4701abe08c995183d023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.138718 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 56d7760f605944c2886d0471c190d3af (DEFAULT_TABLE table=default.Table_888cc00019e343279dac7386f384975b [id=5e0db6c1b00d446d8366e36e146776df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.139046 2081 tablet_service.cc:1505] Processing CreateTablet for tablet e62f956c9f83488e85dda7b99480a7ae (DEFAULT_TABLE table=default.Table_f98c3b707e00420bbd20263069de3dfd [id=c064995ef85e41ae83c79dd5602d68f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.139346 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 84b6e38826b7431fbcf1cb1afef6f641 (DEFAULT_TABLE table=default.Table_072c979a175d433b8ee402eeba321771 [id=7508ed4083eb4ddaa679a6c98a4e2722]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.140079 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43f10bd71013475baf8c62bbb3885bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.141914 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3865399cf6934bc18e16ae2cb4887197
I20251026 02:56:45.143581 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 6ce4092b7ccc4888b5cbe3b38bee2e36
I20251026 02:56:45.147557 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f29197346704f84ba6e3dcb2f5b8bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.148729 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27da2b726344153b54c0187660ca39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.157047 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 780) took 6469 ms. Trace:
I20251026 02:56:45.157186 2372 rpcz_store.cc:276] 1026 02:56:38.687565 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.687620 (+ 55us) service_pool.cc:225] Handling call
1026 02:56:45.017517 (+6329897us) ts_tablet_manager.cc:1151] Deleting tablet f437d8b5f5784e8fbec10d0700e90348
1026 02:56:45.017533 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.022731 (+ 5198us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.116471 (+ 93740us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.157015 (+ 40544us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6329710}
I20251026 02:56:45.159474 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 93cb3875feb744f7bbe72bfdb5c99cb8 (DEFAULT_TABLE table=default.Table_df21327a17374d5f857a5f55cc511c52 [id=753bfe0908c5475692bb35110b436e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.159914 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 12b1795103c1422986e1cdbf0ffdf244 (DEFAULT_TABLE table=default.Table_6f950e766d4c46d4be2009b0c46bcfb4 [id=7446f11a93d5473cae664ca5a51ebb62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.160567 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 4019c3563909495bbc76259b4153e892 (DEFAULT_TABLE table=default.Table_2cfec52f9de94b2096a85e6c32337bcd [id=ef4e90ad64914fc49625732bca76ff49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.161422 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d75ea3e7a6c4aadb438f9e24d9d6a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.162292 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3865399cf6934bc18e16ae2cb4887197. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.162611 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a02c9d84f0ec498e896ec97a75e3c154. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.163094 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 2d75ea3e7a6c4aadb438f9e24d9d6a48 (DEFAULT_TABLE table=default.Table_00a6c982df6b4e6892bc060e529d5ee4 [id=9ce8a1e081c04d8d98c4e90f0e467e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.163717 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e62f956c9f83488e85dda7b99480a7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.164297 2372 tablet_replica.cc:333] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:45.164505 2372 raft_consensus.cc:2243] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:45.164671 2372 raft_consensus.cc:2272] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.166960 2372 ts_tablet_manager.cc:1916] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.167793 2094 tablet_service.cc:1505] Processing CreateTablet for tablet f80b5c4c4e2d4c14a8a5292a05127bc1 (DEFAULT_TABLE table=default.Table_431d446f4c7b4a7b85f4386e28da2d3b [id=727d6a6353ee4c54ab35a6a9b8a8e41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.169235 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d75ea3e7a6c4aadb438f9e24d9d6a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.169124 2240 tablet_service.cc:1505] Processing CreateTablet for tablet a02c9d84f0ec498e896ec97a75e3c154 (DEFAULT_TABLE table=default.Table_2d20995352a745b1a63b0cb745126074 [id=2f89068c1a7b4989b4e5faa8e1cc19c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.169498 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 3865399cf6934bc18e16ae2cb4887197 (DEFAULT_TABLE table=default.Table_2b3b66c0fb3a4f10bf684d40855009d7 [id=bd7ee261e0154f0195abc99ca9790a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.169652 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a02c9d84f0ec498e896ec97a75e3c154. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.169972 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 93cb3875feb744f7bbe72bfdb5c99cb8 (DEFAULT_TABLE table=default.Table_df21327a17374d5f857a5f55cc511c52 [id=753bfe0908c5475692bb35110b436e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.170192 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 2e26156d5a154c08afbd8b1e25a4dfe7 (DEFAULT_TABLE table=default.Table_9b0e2338f40441e6b644b1e7740df97b [id=e47c89a8209646f4bb0e8cd5057a8a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.170545 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 917851a613af4ae3a6e09b4be14447e0 (DEFAULT_TABLE table=default.Table_5b06ba4234b24bf59077dcc622039de2 [id=2abf870df25741b49e289e3ec9d9d474]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.170871 2228 tablet_service.cc:1505] Processing CreateTablet for tablet f27da2b726344153b54c0187660ca39f (DEFAULT_TABLE table=default.Table_016a71d765444bc78bf9e3cf36571b0b [id=cd1317673c124e7a9a9643b45b2ebd3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.171181 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 93d8db69c285430687f5e99110f4bfad (DEFAULT_TABLE table=default.Table_3478623ef36e4ef7910a71260ba0ecb4 [id=bb1263d8e28f4701abe08c995183d023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.172375 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3865399cf6934bc18e16ae2cb4887197. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.173756 2377 ts_tablet_manager.cc:1929] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.175930 2377 log.cc:1199] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/e701d4023fee45d18c48997aab3227c2
I20251026 02:56:45.178530 2381 raft_consensus.cc:359] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.178697 2381 raft_consensus.cc:385] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.178742 2381 raft_consensus.cc:740] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.178936 2381 consensus_queue.cc:260] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.179351 2381 ts_tablet_manager.cc:1434] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.045s user 0.001s sys 0.001s
I20251026 02:56:45.180101 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e26156d5a154c08afbd8b1e25a4dfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.187458 3890 consensus_queue.cc:1048] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:45.191334 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b1795103c1422986e1cdbf0ffdf244. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.192124 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e26156d5a154c08afbd8b1e25a4dfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.192802 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 917851a613af4ae3a6e09b4be14447e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.193421 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb1ec261247744c3bde2ea429e13b19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.195515 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56d7760f605944c2886d0471c190d3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.196498 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b6e38826b7431fbcf1cb1afef6f641. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.197180 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93cb3875feb744f7bbe72bfdb5c99cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.197279 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4019c3563909495bbc76259b4153e892. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.197929 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d8db69c285430687f5e99110f4bfad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.198269 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93cb3875feb744f7bbe72bfdb5c99cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.198549 2381 tablet_bootstrap.cc:492] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:45.198710 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f80b5c4c4e2d4c14a8a5292a05127bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.199256 2377 ts_tablet_manager.cc:1950] T e701d4023fee45d18c48997aab3227c2 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:45.199738 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 862) took 4883 ms. Trace:
I20251026 02:56:45.199932 2381 tablet_bootstrap.cc:654] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.199836 2377 rpcz_store.cc:276] 1026 02:56:40.316543 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.330520 (+ 13977us) service_pool.cc:225] Handling call
1026 02:56:45.089033 (+4758513us) ts_tablet_manager.cc:1151] Deleting tablet e701d4023fee45d18c48997aab3227c2
1026 02:56:45.089044 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.107386 (+ 18342us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.172629 (+ 65243us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.199720 (+ 27091us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4758303}
I20251026 02:56:45.200245 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 861) took 4883 ms. Trace:
I20251026 02:56:45.200310 2377 rpcz_store.cc:276] 1026 02:56:40.316430 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.342699 (+ 26269us) service_pool.cc:225] Handling call
1026 02:56:45.200097 (+4857398us) ts_tablet_manager.cc:1151] Deleting tablet d9391e7654c442a5bb312f0daaf1d99b
1026 02:56:45.200107 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.200215 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4856373}
I20251026 02:56:45.200919 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d8db69c285430687f5e99110f4bfad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.202170 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 917851a613af4ae3a6e09b4be14447e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.203236 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27da2b726344153b54c0187660ca39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.204396 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 863) took 4830 ms. Trace:
I20251026 02:56:45.204486 2377 rpcz_store.cc:276] 1026 02:56:40.373854 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.373911 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:45.204287 (+4830376us) ts_tablet_manager.cc:1151] Deleting tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61
1026 02:56:45.204296 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.204383 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4830212}
I20251026 02:56:45.204788 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 864) took 4829 ms. Trace:
I20251026 02:56:45.204845 2377 rpcz_store.cc:276] 1026 02:56:40.375775 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.375823 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:45.204697 (+4828874us) ts_tablet_manager.cc:1151] Deleting tablet 45e79be9b98945c49fb712019b21b91b
1026 02:56:45.204706 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.204777 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4828727}
I20251026 02:56:45.205196 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 865) took 4827 ms. Trace:
I20251026 02:56:45.205256 2377 rpcz_store.cc:276] 1026 02:56:40.377323 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.377373 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:45.205054 (+4827681us) ts_tablet_manager.cc:1151] Deleting tablet 4e645650709f47eea4243ab64ec31312
1026 02:56:45.205062 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.205183 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4827470}
I20251026 02:56:45.205526 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 866) took 4815 ms. Trace:
I20251026 02:56:45.133747 2378 tablet_bootstrap.cc:654] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.136467 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 2d75ea3e7a6c4aadb438f9e24d9d6a48 (DEFAULT_TABLE table=default.Table_00a6c982df6b4e6892bc060e529d5ee4 [id=9ce8a1e081c04d8d98c4e90f0e467e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.206375 2376 log.cc:1199] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/17d2cc6b16d34331bb80c576c0cd2ce0
I20251026 02:56:45.136749 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 93cb3875feb744f7bbe72bfdb5c99cb8 (DEFAULT_TABLE table=default.Table_df21327a17374d5f857a5f55cc511c52 [id=753bfe0908c5475692bb35110b436e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.206630 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d75ea3e7a6c4aadb438f9e24d9d6a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.206738 2376 ts_tablet_manager.cc:1950] T 17d2cc6b16d34331bb80c576c0cd2ce0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:45.137080 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 2e26156d5a154c08afbd8b1e25a4dfe7 (DEFAULT_TABLE table=default.Table_9b0e2338f40441e6b644b1e7740df97b [id=e47c89a8209646f4bb0e8cd5057a8a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.206931 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93cb3875feb744f7bbe72bfdb5c99cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.207036 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 793) took 5337 ms. Trace:
I20251026 02:56:45.207140 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e26156d5a154c08afbd8b1e25a4dfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.207121 2376 rpcz_store.cc:276] 1026 02:56:39.869479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.869550 (+ 71us) service_pool.cc:225] Handling call
1026 02:56:44.621476 (+4751926us) ts_tablet_manager.cc:1151] Deleting tablet 17d2cc6b16d34331bb80c576c0cd2ce0
1026 02:56:44.621484 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:44.858060 (+236576us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.127813 (+269753us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.207018 (+ 79205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4751400}
I20251026 02:56:45.208665 2096 tablet_service.cc:1505] Processing CreateTablet for tablet d44f26dba69c43ee91cb10fb96f6a802 (DEFAULT_TABLE table=default.Table_df944b7ec6554ea687b16b286ba44c55 [id=ce9503986635482c8396d7f77ce2a66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.209088 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d44f26dba69c43ee91cb10fb96f6a802. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.131227 3988 raft_consensus.cc:493] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.209738 3988 raft_consensus.cc:515] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.210901 2376 tablet_replica.cc:333] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:45.211091 2376 raft_consensus.cc:2243] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:45.211258 2376 raft_consensus.cc:2272] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.211803 2376 ts_tablet_manager.cc:1916] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.137951 1943 tablet_service.cc:1505] Processing CreateTablet for tablet a02c9d84f0ec498e896ec97a75e3c154 (DEFAULT_TABLE table=default.Table_2d20995352a745b1a63b0cb745126074 [id=2f89068c1a7b4989b4e5faa8e1cc19c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.212466 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a02c9d84f0ec498e896ec97a75e3c154. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.213383 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5b3d81865782468ca8ab3c0023b31352"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:45.214028 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b3d81865782468ca8ab3c0023b31352 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:45.214216 3849 raft_consensus.cc:493] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.214332 3849 raft_consensus.cc:515] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.138302 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 3865399cf6934bc18e16ae2cb4887197 (DEFAULT_TABLE table=default.Table_2b3b66c0fb3a4f10bf684d40855009d7 [id=bd7ee261e0154f0195abc99ca9790a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.214941 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3865399cf6934bc18e16ae2cb4887197. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.138907 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 917851a613af4ae3a6e09b4be14447e0 (DEFAULT_TABLE table=default.Table_5b06ba4234b24bf59077dcc622039de2 [id=2abf870df25741b49e289e3ec9d9d474]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.221490 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 917851a613af4ae3a6e09b4be14447e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.139159 1951 tablet_service.cc:1505] Processing CreateTablet for tablet f27da2b726344153b54c0187660ca39f (DEFAULT_TABLE table=default.Table_016a71d765444bc78bf9e3cf36571b0b [id=cd1317673c124e7a9a9643b45b2ebd3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.224737 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db275efdd344817b11a2168675cf95e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:45.139492 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 93d8db69c285430687f5e99110f4bfad (DEFAULT_TABLE table=default.Table_3478623ef36e4ef7910a71260ba0ecb4 [id=bb1263d8e28f4701abe08c995183d023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.139797 1944 tablet_service.cc:1505] Processing CreateTablet for tablet bb1ec261247744c3bde2ea429e13b19d (DEFAULT_TABLE table=default.Table_9dc1a989050c4cf48b9a8111a722b2ee [id=ddcd017fd0dd4fbdbacdcd370b9085c3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:45.225226 2053 leader_election.cc:343] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.228312 3988 leader_election.cc:290] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.232112 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27da2b726344153b54c0187660ca39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.232749 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb1ec261247744c3bde2ea429e13b19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.233160 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d8db69c285430687f5e99110f4bfad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.234015 3849 leader_election.cc:290] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.234333 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a1442fffe54add93666e3c78a0cc58" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:45.231760 2381 tablet_bootstrap.cc:492] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
W20251026 02:56:45.234941 2053 leader_election.cc:343] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.235003 2381 ts_tablet_manager.cc:1403] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.002s
I20251026 02:56:45.236394 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a1442fffe54add93666e3c78a0cc58" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:45.237250 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db275efdd344817b11a2168675cf95e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:45.236985 2053 leader_election.cc:343] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.238442 2053 leader_election.cc:304] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:45.239121 2375 tablet_bootstrap.cc:492] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:45.239248 2375 ts_tablet_manager.cc:1403] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.232s user 0.011s sys 0.003s
I20251026 02:56:45.239667 2381 raft_consensus.cc:359] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.239807 2381 raft_consensus.cc:385] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.239784 3849 raft_consensus.cc:2749] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:45.239847 2381 raft_consensus.cc:740] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.240118 2381 consensus_queue.cc:260] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.240655 2259 raft_consensus.cc:2468] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:45.243279 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 56d7760f605944c2886d0471c190d3af (DEFAULT_TABLE table=default.Table_888cc00019e343279dac7386f384975b [id=5e0db6c1b00d446d8366e36e146776df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.243674 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56d7760f605944c2886d0471c190d3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.248658 2381 ts_tablet_manager.cc:1434] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.014s user 0.000s sys 0.001s
I20251026 02:56:45.248958 2381 tablet_bootstrap.cc:492] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:45.249482 3849 raft_consensus.cc:493] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.249606 3849 raft_consensus.cc:515] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.249727 2234 tablet_service.cc:1505] Processing CreateTablet for tablet bb1ec261247744c3bde2ea429e13b19d (DEFAULT_TABLE table=default.Table_9dc1a989050c4cf48b9a8111a722b2ee [id=ddcd017fd0dd4fbdbacdcd370b9085c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.250052 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb1ec261247744c3bde2ea429e13b19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.250401 2053 leader_election.cc:304] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:45.258001 2381 tablet_bootstrap.cc:654] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.248087 2375 raft_consensus.cc:359] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.263127 2375 raft_consensus.cc:385] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.263211 2375 raft_consensus.cc:740] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.263425 2375 consensus_queue.cc:260] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.263990 2375 ts_tablet_manager.cc:1434] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20251026 02:56:45.264276 2375 tablet_bootstrap.cc:492] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:45.266117 3841 raft_consensus.cc:2804] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:45.266201 3841 raft_consensus.cc:493] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:45.266263 3841 raft_consensus.cc:3060] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:45.266731 3849 leader_election.cc:290] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.263640 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 12b1795103c1422986e1cdbf0ffdf244 (DEFAULT_TABLE table=default.Table_6f950e766d4c46d4be2009b0c46bcfb4 [id=7446f11a93d5473cae664ca5a51ebb62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.276541 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b1795103c1422986e1cdbf0ffdf244. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.283246 2229 tablet_service.cc:1505] Processing CreateTablet for tablet e62f956c9f83488e85dda7b99480a7ae (DEFAULT_TABLE table=default.Table_f98c3b707e00420bbd20263069de3dfd [id=c064995ef85e41ae83c79dd5602d68f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.283681 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e62f956c9f83488e85dda7b99480a7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.285104 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 533c417721d1455985de4368fafcf145 (DEFAULT_TABLE table=default.Table_e6801c439a1b498c82d443a4e2fc8e74 [id=3bb599754ee84994907c9ec80ab96095]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.285558 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533c417721d1455985de4368fafcf145. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.285918 2100 tablet_service.cc:1505] Processing CreateTablet for tablet b7b5619d7b7243a99aff0c5fb29e499e (DEFAULT_TABLE table=default.Table_f78aa520baff4e07be8f8a125262a3bd [id=29902fec62794f01afe51e4989ba95dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.286141 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 4019c3563909495bbc76259b4153e892 (DEFAULT_TABLE table=default.Table_2cfec52f9de94b2096a85e6c32337bcd [id=ef4e90ad64914fc49625732bca76ff49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.287132 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 4019c3563909495bbc76259b4153e892 (DEFAULT_TABLE table=default.Table_2cfec52f9de94b2096a85e6c32337bcd [id=ef4e90ad64914fc49625732bca76ff49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.287472 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4019c3563909495bbc76259b4153e892. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.288880 2231 tablet_service.cc:1505] Processing CreateTablet for tablet f80b5c4c4e2d4c14a8a5292a05127bc1 (DEFAULT_TABLE table=default.Table_431d446f4c7b4a7b85f4386e28da2d3b [id=727d6a6353ee4c54ab35a6a9b8a8e41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.289300 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f80b5c4c4e2d4c14a8a5292a05127bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.289548 1954 tablet_service.cc:1505] Processing CreateTablet for tablet d44f26dba69c43ee91cb10fb96f6a802 (DEFAULT_TABLE table=default.Table_df944b7ec6554ea687b16b286ba44c55 [id=ce9503986635482c8396d7f77ce2a66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.290423 2233 tablet_service.cc:1505] Processing CreateTablet for tablet d44f26dba69c43ee91cb10fb96f6a802 (DEFAULT_TABLE table=default.Table_df944b7ec6554ea687b16b286ba44c55 [id=ce9503986635482c8396d7f77ce2a66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.290812 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d44f26dba69c43ee91cb10fb96f6a802. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.291302 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 533c417721d1455985de4368fafcf145 (DEFAULT_TABLE table=default.Table_e6801c439a1b498c82d443a4e2fc8e74 [id=3bb599754ee84994907c9ec80ab96095]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.291760 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4019c3563909495bbc76259b4153e892. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.291971 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d44f26dba69c43ee91cb10fb96f6a802. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.298331 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533c417721d1455985de4368fafcf145. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.288508 1944 tablet_service.cc:1505] Processing CreateTablet for tablet f80b5c4c4e2d4c14a8a5292a05127bc1 (DEFAULT_TABLE table=default.Table_431d446f4c7b4a7b85f4386e28da2d3b [id=727d6a6353ee4c54ab35a6a9b8a8e41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.300658 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f80b5c4c4e2d4c14a8a5292a05127bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.302470 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 56d7760f605944c2886d0471c190d3af (DEFAULT_TABLE table=default.Table_888cc00019e343279dac7386f384975b [id=5e0db6c1b00d446d8366e36e146776df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.302845 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56d7760f605944c2886d0471c190d3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.304534 1951 tablet_service.cc:1505] Processing CreateTablet for tablet e62f956c9f83488e85dda7b99480a7ae (DEFAULT_TABLE table=default.Table_f98c3b707e00420bbd20263069de3dfd [id=c064995ef85e41ae83c79dd5602d68f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.304977 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e62f956c9f83488e85dda7b99480a7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.305539 2227 tablet_service.cc:1505] Processing CreateTablet for tablet 533c417721d1455985de4368fafcf145 (DEFAULT_TABLE table=default.Table_e6801c439a1b498c82d443a4e2fc8e74 [id=3bb599754ee84994907c9ec80ab96095]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.305869 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533c417721d1455985de4368fafcf145. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.305871 1943 tablet_service.cc:1505] Processing CreateTablet for tablet b7b5619d7b7243a99aff0c5fb29e499e (DEFAULT_TABLE table=default.Table_f78aa520baff4e07be8f8a125262a3bd [id=29902fec62794f01afe51e4989ba95dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.306156 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b5619d7b7243a99aff0c5fb29e499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.309577 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 12b1795103c1422986e1cdbf0ffdf244 (DEFAULT_TABLE table=default.Table_6f950e766d4c46d4be2009b0c46bcfb4 [id=7446f11a93d5473cae664ca5a51ebb62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.309926 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b1795103c1422986e1cdbf0ffdf244. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.312618 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 84b6e38826b7431fbcf1cb1afef6f641 (DEFAULT_TABLE table=default.Table_072c979a175d433b8ee402eeba321771 [id=7508ed4083eb4ddaa679a6c98a4e2722]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.312937 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b6e38826b7431fbcf1cb1afef6f641. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.313928 2375 tablet_bootstrap.cc:654] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.318655 2233 tablet_service.cc:1505] Processing CreateTablet for tablet b7b5619d7b7243a99aff0c5fb29e499e (DEFAULT_TABLE table=default.Table_f78aa520baff4e07be8f8a125262a3bd [id=29902fec62794f01afe51e4989ba95dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.318889 2083 tablet_service.cc:1552] Processing DeleteTablet for tablet 3865399cf6934bc18e16ae2cb4887197 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:44 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:45.319028 2083 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:44 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:45.319047 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b5619d7b7243a99aff0c5fb29e499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.321702 2235 tablet_service.cc:1505] Processing CreateTablet for tablet f105398b04454473b2a283e729eb2139 (DEFAULT_TABLE table=default.Table_37167b3e100a458d81c0578a964bee27 [id=25d2c9fa6b7f4f23bdae7d8f81278295]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.322093 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f105398b04454473b2a283e729eb2139. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.324355 2082 tablet_service.cc:1505] Processing CreateTablet for tablet f105398b04454473b2a283e729eb2139 (DEFAULT_TABLE table=default.Table_37167b3e100a458d81c0578a964bee27 [id=25d2c9fa6b7f4f23bdae7d8f81278295]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.336607 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet f437d8b5f5784e8fbec10d0700e90348 (table default.Table_b555f404492a47808d8a9a168a45ef2f [id=6185bafc43f340dfa6e187a6e5468874]) successfully deleted
W20251026 02:56:45.336830 1648 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 52327.1us
I20251026 02:56:45.338696 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 3865399cf6934bc18e16ae2cb4887197 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:44 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.345425 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:44 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.347333 2227 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:44 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:45.349207 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 already in progress: creating tablet
I20251026 02:56:45.350844 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 3865399cf6934bc18e16ae2cb4887197 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:44 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:45.354806 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 84b6e38826b7431fbcf1cb1afef6f641 (DEFAULT_TABLE table=default.Table_072c979a175d433b8ee402eeba321771 [id=7508ed4083eb4ddaa679a6c98a4e2722]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.355253 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b6e38826b7431fbcf1cb1afef6f641. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:45.355566 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 45e79be9b98945c49fb712019b21b91b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45e79be9b98945c49fb712019b21b91b already in progress: creating tablet
W20251026 02:56:45.356537 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 4e645650709f47eea4243ab64ec31312 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e645650709f47eea4243ab64ec31312 already in progress: creating tablet
I20251026 02:56:45.361048 2376 ts_tablet_manager.cc:1929] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.366883 2376 log.cc:1199] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/e701d4023fee45d18c48997aab3227c2
W20251026 02:56:45.367096 2194 consensus_peers.cc:597] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:45.368706 2376 ts_tablet_manager.cc:1950] T e701d4023fee45d18c48997aab3227c2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:45.369292 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 17d2cc6b16d34331bb80c576c0cd2ce0 (table default.Table_9f3815b1bf464a83a574e3075d065956 [id=095ad208529348d4aea2d4e2840e07c7]) successfully deleted
I20251026 02:56:45.372740 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 816) took 5045 ms. Trace:
I20251026 02:56:45.377089 2376 rpcz_store.cc:276] 1026 02:56:40.327457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.328038 (+ 581us) service_pool.cc:225] Handling call
1026 02:56:45.210853 (+4882815us) ts_tablet_manager.cc:1151] Deleting tablet e701d4023fee45d18c48997aab3227c2
1026 02:56:45.210865 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.334172 (+123307us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.360910 (+ 26738us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.372713 (+ 11803us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4882631}
I20251026 02:56:45.381839 2376 tablet_replica.cc:333] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:45.382406 2376 raft_consensus.cc:2243] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:45.383845 2376 raft_consensus.cc:2272] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.387365 2376 ts_tablet_manager.cc:1916] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.391048 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 4e1291f47f384ff38eadd16040db65b8
I20251026 02:56:45.393577 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ac6c5e4166e14e079228d171f79e964c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:45.394286 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac6c5e4166e14e079228d171f79e964c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:45.396700 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 94d5faa4584a4e36b38bdd552d641994
I20251026 02:56:45.397692 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 94d5faa4584a4e36b38bdd552d641994 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:44 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.398104 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_338870d29792464aa2ab8c8c2f5978ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:45.398720 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_338870d29792464aa2ab8c8c2f5978ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:45.402350 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1abe312e689149299920c58ec6f1a35d" } new_table_name: "default.Table_e2177d9c4a5e4c26af6296ef73fb3c34" modify_external_catalogs: true
I20251026 02:56:45.403788 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e1291f47f384ff38eadd16040db65b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:44 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:45.414618 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet 142043b5d558490c9399626c3764ff7e from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:45.415475 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b5619d7b7243a99aff0c5fb29e499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.416742 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f105398b04454473b2a283e729eb2139. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.418690 3841 raft_consensus.cc:515] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.419260 3841 leader_election.cc:290] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.419378 3841 raft_consensus.cc:493] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.419458 3841 raft_consensus.cc:515] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.419850 3841 leader_election.cc:290] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.433194 2376 ts_tablet_manager.cc:1929] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.433336 2376 log.cc:1199] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/44af74033ccf4d048475f12d164938f5
I20251026 02:56:45.433686 2376 ts_tablet_manager.cc:1950] T 44af74033ccf4d048475f12d164938f5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:45.433984 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 815) took 5106 ms. Trace:
I20251026 02:56:45.434093 2376 rpcz_store.cc:276] 1026 02:56:40.327320 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.328883 (+ 1563us) service_pool.cc:225] Handling call
1026 02:56:45.381776 (+5052893us) ts_tablet_manager.cc:1151] Deleting tablet 44af74033ccf4d048475f12d164938f5
1026 02:56:45.381792 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.415403 (+ 33611us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.433071 (+ 17668us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.433964 (+ 893us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5052706}
I20251026 02:56:45.434309 2372 ts_tablet_manager.cc:1929] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.434408 2372 log.cc:1199] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/aa10cd990a944eab8d9eb66aaf20080c
I20251026 02:56:45.434546 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 814) took 5107 ms. Trace:
I20251026 02:56:45.434643 2376 rpcz_store.cc:276] 1026 02:56:40.327231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.329119 (+ 1888us) service_pool.cc:225] Handling call
1026 02:56:45.434396 (+5105277us) ts_tablet_manager.cc:1151] Deleting tablet d9391e7654c442a5bb312f0daaf1d99b
1026 02:56:45.434408 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.434530 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5105097}
I20251026 02:56:45.434725 2372 ts_tablet_manager.cc:1950] T aa10cd990a944eab8d9eb66aaf20080c P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:45.434974 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 817) took 5062 ms. Trace:
I20251026 02:56:45.435047 2376 rpcz_store.cc:276] 1026 02:56:40.372869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.372935 (+ 66us) service_pool.cc:225] Handling call
1026 02:56:45.434876 (+5061941us) ts_tablet_manager.cc:1151] Deleting tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61
1026 02:56:45.434886 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.434960 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5061746}
I20251026 02:56:45.435348 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 44af74033ccf4d048475f12d164938f5 (table default.Table_e4fbc44faf984982986abfb227d6663d [id=b35a8189bbc04e94bb097c9e13a38ec6]) successfully deleted
W20251026 02:56:45.435864 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 already in progress: creating tablet
I20251026 02:56:45.437284 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e1291f47f384ff38eadd16040db65b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:44 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.447180 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 94d5faa4584a4e36b38bdd552d641994 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:44 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:45.451998 1955 tablet_service.cc:1505] Processing CreateTablet for tablet f105398b04454473b2a283e729eb2139 (DEFAULT_TABLE table=default.Table_37167b3e100a458d81c0578a964bee27 [id=25d2c9fa6b7f4f23bdae7d8f81278295]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:45.452555 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f105398b04454473b2a283e729eb2139. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.463351 2378 tablet_bootstrap.cc:492] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:45.463519 2378 ts_tablet_manager.cc:1403] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.398s user 0.004s sys 0.000s
I20251026 02:56:45.464336 2378 raft_consensus.cc:359] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.464551 2378 raft_consensus.cc:385] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.464600 2378 raft_consensus.cc:740] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.464802 2378 consensus_queue.cc:260] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.465286 2378 ts_tablet_manager.cc:1434] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:45.465597 2378 tablet_bootstrap.cc:492] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:45.467200 2378 tablet_bootstrap.cc:654] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.471959 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 818) took 5097 ms. Trace:
I20251026 02:56:45.472080 2376 rpcz_store.cc:276] 1026 02:56:40.374936 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.374998 (+ 62us) service_pool.cc:225] Handling call
1026 02:56:45.471820 (+5096822us) ts_tablet_manager.cc:1151] Deleting tablet 45e79be9b98945c49fb712019b21b91b
1026 02:56:45.471833 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.471936 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5060072}
I20251026 02:56:45.472529 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 819) took 5096 ms. Trace:
I20251026 02:56:45.473317 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 781) took 6784 ms. Trace:
I20251026 02:56:45.473417 2372 rpcz_store.cc:276] 1026 02:56:38.688450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.688494 (+ 44us) service_pool.cc:225] Handling call
1026 02:56:45.157450 (+6468956us) ts_tablet_manager.cc:1151] Deleting tablet aa10cd990a944eab8d9eb66aaf20080c
1026 02:56:45.164218 (+ 6768us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.326227 (+162009us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.434206 (+107979us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.473290 (+ 39084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6468804}
I20251026 02:56:45.473796 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 806) took 6520 ms. Trace:
I20251026 02:56:45.473898 2372 rpcz_store.cc:276] 1026 02:56:38.953762 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.953948 (+ 186us) service_pool.cc:225] Handling call
1026 02:56:45.473675 (+6519727us) ts_tablet_manager.cc:1151] Deleting tablet 46547883cfe041f3954aa4128fba1d8c
1026 02:56:45.473686 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.473780 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6519539}
I20251026 02:56:45.474233 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 808) took 6518 ms. Trace:
I20251026 02:56:45.474300 2372 rpcz_store.cc:276] 1026 02:56:38.955534 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.955580 (+ 46us) service_pool.cc:225] Handling call
1026 02:56:45.474099 (+6518519us) ts_tablet_manager.cc:1151] Deleting tablet 3e424a9c89e64b1c98ada17fa5c31e9e
1026 02:56:45.474108 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.474192 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6518326}
I20251026 02:56:45.474579 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 807) took 6520 ms. Trace:
I20251026 02:56:45.474639 2372 rpcz_store.cc:276] 1026 02:56:38.953905 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.956758 (+ 2853us) service_pool.cc:225] Handling call
1026 02:56:45.474492 (+6517734us) ts_tablet_manager.cc:1151] Deleting tablet 02861f399f4d488586f279c01eebcdd1
1026 02:56:45.474499 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.474566 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6517476}
I20251026 02:56:45.474915 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 824) took 6373 ms. Trace:
I20251026 02:56:45.474975 2372 rpcz_store.cc:276] 1026 02:56:39.101846 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.102207 (+ 361us) service_pool.cc:225] Handling call
1026 02:56:45.474834 (+6372627us) ts_tablet_manager.cc:1151] Deleting tablet b819b43894574dc9ad52b4081e46628e
1026 02:56:45.474842 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.474903 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6372393}
I20251026 02:56:45.475231 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 825) took 6345 ms. Trace:
I20251026 02:56:45.475317 2372 rpcz_store.cc:276] 1026 02:56:39.129305 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.185407 (+ 56102us) service_pool.cc:225] Handling call
1026 02:56:45.475134 (+6289727us) ts_tablet_manager.cc:1151] Deleting tablet 7053d454def04264a956aa75ceaad5c9
1026 02:56:45.475141 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.475218 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6289558}
I20251026 02:56:45.475328 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0e0f079c121342b9bb342f26e3c24518" } modify_external_catalogs: true
I20251026 02:56:45.475539 2372 tablet_replica.cc:333] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:45.475523 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0e0f079c121342b9bb342f26e3c24518" } modify_external_catalogs: true
I20251026 02:56:45.476271 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet bcd5621aa3754733b22508be89f080db from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:45.476611 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 45e79be9b98945c49fb712019b21b91b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45e79be9b98945c49fb712019b21b91b already in progress: creating tablet
I20251026 02:56:45.476763 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 53264ef08ff7454faa19b18fa35bd803 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:45.477663 3841 raft_consensus.cc:493] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.477836 3841 raft_consensus.cc:515] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.478412 3841 leader_election.cc:290] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.485332 2027 heartbeater.cc:344] Connected to a master server at 127.1.150.60:33729
I20251026 02:56:45.490959 2376 rpcz_store.cc:276] 1026 02:56:40.376415 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.376499 (+ 84us) service_pool.cc:225] Handling call
1026 02:56:45.472407 (+5095908us) ts_tablet_manager.cc:1151] Deleting tablet 4e645650709f47eea4243ab64ec31312
1026 02:56:45.472417 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.472513 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5095689}
I20251026 02:56:45.491535 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 821) took 5101 ms. Trace:
W20251026 02:56:45.491714 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 4e645650709f47eea4243ab64ec31312 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e645650709f47eea4243ab64ec31312 already in progress: creating tablet
I20251026 02:56:45.496649 2376 rpcz_store.cc:276] 1026 02:56:40.390576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.390644 (+ 68us) service_pool.cc:225] Handling call
1026 02:56:45.491381 (+5100737us) ts_tablet_manager.cc:1151] Deleting tablet 9b9a5daa36b547a7888d38dc3329eaf2
1026 02:56:45.491393 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.491515 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5076735}
I20251026 02:56:45.497216 3893 consensus_queue.cc:1048] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:45.499902 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 882cd1680aa140f6a63a961cd4aec25a (table default.Table_52417f25dbdb4676a5aaaca9bf46962c [id=cc7a8c37a3b342d988872e3baa16a26d]). Expected version 1 got 0
I20251026 02:56:45.500129 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 2fd8f02126b24285bbfa575540ba0f09 (table default.Table_7539d36a088f4eafaa21d151ed9ea1cb [id=1950d7cb826b4a99b1ba8866e1d845f3]). Expected version 1 got 0
I20251026 02:56:45.500801 1662 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5] alter complete (version 1)
I20251026 02:56:45.502132 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_64db0f5adb1740ce80b215181ec835fe" } new_table_name: "default.Table_b161497cfeea47febcaaabb994fd8862" modify_external_catalogs: true
I20251026 02:56:45.502565 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 94d5faa4584a4e36b38bdd552d641994 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:44 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:45.503248 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet aa10cd990a944eab8d9eb66aaf20080c (table default.Table_42c4c5e055054536b2b541711288a481 [id=503128cdfe654cfb9713c8e241eac740]) successfully deleted
W20251026 02:56:45.503479 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 46547883cfe041f3954aa4128fba1d8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46547883cfe041f3954aa4128fba1d8c already in progress: creating tablet
W20251026 02:56:45.503674 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3e424a9c89e64b1c98ada17fa5c31e9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e424a9c89e64b1c98ada17fa5c31e9e already in progress: creating tablet
W20251026 02:56:45.503829 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 02861f399f4d488586f279c01eebcdd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02861f399f4d488586f279c01eebcdd1 already in progress: creating tablet
W20251026 02:56:45.503999 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet b819b43894574dc9ad52b4081e46628e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b819b43894574dc9ad52b4081e46628e already in progress: creating tablet
W20251026 02:56:45.504153 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7053d454def04264a956aa75ceaad5c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7053d454def04264a956aa75ceaad5c9 already in progress: creating tablet
I20251026 02:56:45.504936 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet a53f25688e3b45dca68984e2c543d094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:45.509328 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 7801f073d5d94076af9299492f86fad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:26 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:45.509567 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 34c6a7c774044411a4733b115080c4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:45.512460 2225 tablet_service.cc:1552] Processing DeleteTablet for tablet a012d868a78f4324a92db83c81931b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:45.513383 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b7d85a2941545ceaa16b22e8328b4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:45.513072 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet e701d4023fee45d18c48997aab3227c2 with delete_type TABLET_DATA_DELETED (replaced by tablet ac82c14e7dcb448b845d998249e2b339) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:45.514336 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet f437d8b5f5784e8fbec10d0700e90348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:45.514958 3893 raft_consensus.cc:493] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.515128 3893 raft_consensus.cc:515] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.515859 3893 leader_election.cc:290] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:45.518221 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_11cf708e0e8243c887777e42acc25490" } new_table_name: "default.Table_2cf16843709c4e2a9c3632c25c6bbe25" modify_external_catalogs: true
I20251026 02:56:45.524935 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c6a7c774044411a4733b115080c4c6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:45.525154 2114 raft_consensus.cc:2468] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:45.525840 2195 leader_election.cc:304] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:45.526268 3893 raft_consensus.cc:2804] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:45.526338 3893 raft_consensus.cc:493] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:45.526397 3893 raft_consensus.cc:3060] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:45.530287 3893 raft_consensus.cc:515] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.532382 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c6a7c774044411a4733b115080c4c6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:45.532617 2114 raft_consensus.cc:3060] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:45.534912 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 9b9a5daa36b547a7888d38dc3329eaf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b9a5daa36b547a7888d38dc3329eaf2 already in progress: creating tablet
I20251026 02:56:45.526648 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet bfa68f2c942c43e2a02e9d333bdfa2a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 262f60fdb274476abf3533b4f47688b8) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.526835 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet d212170f056a44dfb15cfcb7f9fbae2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.535576 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 825) took 5121 ms. Trace:
I20251026 02:56:45.535643 2114 raft_consensus.cc:2468] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:45.535678 2376 rpcz_store.cc:276] 1026 02:56:40.414382 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.414455 (+ 73us) service_pool.cc:225] Handling call
1026 02:56:45.535393 (+5120938us) ts_tablet_manager.cc:1151] Deleting tablet 7096b78402074049b9c8ed1b9fe21f0e
1026 02:56:45.535407 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.535551 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23,"tablet-delete.queue_time_us":5120739}
W20251026 02:56:45.536303 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7096b78402074049b9c8ed1b9fe21f0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7096b78402074049b9c8ed1b9fe21f0e already in progress: creating tablet
I20251026 02:56:45.536497 2195 leader_election.cc:304] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:45.537489 3984 raft_consensus.cc:493] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.537614 3984 raft_consensus.cc:515] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.538189 3984 leader_election.cc:290] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.538846 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 820) took 5150 ms. Trace:
I20251026 02:56:45.539110 4003 raft_consensus.cc:2804] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:45.541074 3893 leader_election.cc:290] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:45.541392 4003 raft_consensus.cc:697] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:45.542126 4003 consensus_queue.cc:237] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.540273 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f437d8b5f5784e8fbec10d0700e90348" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:45.549108 2381 tablet_bootstrap.cc:492] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:45.549364 2381 ts_tablet_manager.cc:1403] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.300s user 0.002s sys 0.005s
I20251026 02:56:45.549876 2375 tablet_bootstrap.cc:492] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:45.550202 2381 raft_consensus.cc:359] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.550485 2381 raft_consensus.cc:385] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.550598 2381 raft_consensus.cc:740] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.550899 2381 consensus_queue.cc:260] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.551625 2381 ts_tablet_manager.cc:1434] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:45.552006 2381 tablet_bootstrap.cc:492] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:45.552021 2375 ts_tablet_manager.cc:1403] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.288s user 0.007s sys 0.006s
I20251026 02:56:45.552479 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet 885c10070d6942f7abe52c4558d887e6 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:45.553031 2375 raft_consensus.cc:359] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.553215 2375 raft_consensus.cc:385] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.553278 2375 raft_consensus.cc:740] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.550297 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e1291f47f384ff38eadd16040db65b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:44 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:45.553115 2372 raft_consensus.cc:2243] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:45.553514 2381 tablet_bootstrap.cc:654] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.553645 2372 raft_consensus.cc:2272] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.554172 2372 ts_tablet_manager.cc:1916] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.554563 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a53f25688e3b45dca68984e2c543d094" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:45.555123 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db275efdd344817b11a2168675cf95e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:45.555250 2259 raft_consensus.cc:3060] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:45.557940 2259 raft_consensus.cc:2468] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:45.558672 3997 raft_consensus.cc:493] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.558806 3997 raft_consensus.cc:515] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.559095 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48105e54d6324622a032e1288a46188c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:45.559341 3997 leader_election.cc:290] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.560654 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c7efb70c0804c149f36440a4a9e6ef8" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:45.560883 2259 raft_consensus.cc:2468] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:45.561321 2372 ts_tablet_manager.cc:1929] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.561437 2372 log.cc:1199] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/21b936fa4f6a4ff6af321971a30cccbe
I20251026 02:56:45.561832 2372 ts_tablet_manager.cc:1950] T 21b936fa4f6a4ff6af321971a30cccbe P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:45.562161 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 826) took 6390 ms. Trace:
I20251026 02:56:45.562266 2372 rpcz_store.cc:276] 1026 02:56:39.171931 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.188885 (+ 16954us) service_pool.cc:225] Handling call
1026 02:56:45.475497 (+6286612us) ts_tablet_manager.cc:1151] Deleting tablet 21b936fa4f6a4ff6af321971a30cccbe
1026 02:56:45.475507 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.557928 (+ 82421us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.561197 (+ 3269us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.562143 (+ 946us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77375,"tablet-delete.queue_time_us":6262946}
I20251026 02:56:45.563355 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 827) took 6174 ms. Trace:
I20251026 02:56:45.563436 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 21b936fa4f6a4ff6af321971a30cccbe (table default.Table_4e2e1ce85b8246388602db5ddc857f79 [id=5fb61e88f7074976bac29b40f4466ecb]) successfully deleted
I20251026 02:56:45.563452 2372 rpcz_store.cc:276] 1026 02:56:39.389103 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.389162 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:45.563236 (+6174074us) ts_tablet_manager.cc:1151] Deleting tablet be1c0948c0ae4563b7cb644521914cb9
1026 02:56:45.563250 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.563335 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6173838}
I20251026 02:56:45.563712 2372 tablet_replica.cc:333] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:45.563947 2372 raft_consensus.cc:2243] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:45.564142 2372 raft_consensus.cc:2272] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:45.564266 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet be1c0948c0ae4563b7cb644521914cb9 because the tablet was not found. No further retry: Not found: Tablet not found: be1c0948c0ae4563b7cb644521914cb9
I20251026 02:56:45.564953 2372 ts_tablet_manager.cc:1916] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.565461 2375 consensus_queue.cc:260] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.570503 2375 ts_tablet_manager.cc:1434] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.018s user 0.000s sys 0.001s
I20251026 02:56:45.571393 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b9a5daa36b547a7888d38dc3329eaf2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:45.539253 1916 leader_election.cc:343] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: f437d8b5f5784e8fbec10d0700e90348
I20251026 02:56:45.551209 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a53f25688e3b45dca68984e2c543d094" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:45.577708 2376 rpcz_store.cc:276] 1026 02:56:40.388872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.388933 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:45.538724 (+5149791us) ts_tablet_manager.cc:1151] Deleting tablet 8a076043f05c42d9afd9699fb4bec5ba
1026 02:56:45.538735 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.538829 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5123812}
I20251026 02:56:45.551663 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db275efdd344817b11a2168675cf95e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:45.554240 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c7efb70c0804c149f36440a4a9e6ef8" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:45.556483 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48105e54d6324622a032e1288a46188c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:45.578425 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 8a076043f05c42d9afd9699fb4bec5ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a076043f05c42d9afd9699fb4bec5ba already in progress: creating tablet
I20251026 02:56:45.578853 1915 leader_election.cc:304] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:45.579583 3984 raft_consensus.cc:2749] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:45.580053 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c6a7c774044411a4733b115080c4c6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:45.580192 2376 tablet_replica.cc:333] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:45.580411 2376 raft_consensus.cc:2243] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:45.580587 2376 raft_consensus.cc:2272] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.580703 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c6a7c774044411a4733b115080c4c6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:45.581337 2194 leader_election.cc:343] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:45.581755 2194 consensus_peers.cc:597] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:45.581902 2194 leader_election.cc:343] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.568462 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 21b936fa4f6a4ff6af321971a30cccbe with delete_type TABLET_DATA_DELETED (replaced by tablet a956ebfb1eaa4d6fb1575b4165293193) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.568925 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet f437d8b5f5784e8fbec10d0700e90348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.591220 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3ff405b3d7584fa9b587d962606e80c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:45.569928 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b9a5daa36b547a7888d38dc3329eaf2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:45.592515 2195 consensus_peers.cc:597] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7177931353164b2aa82c00b2807ccd63. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:45.572948 2378 tablet_bootstrap.cc:492] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:45.591466 1982 raft_consensus.cc:3060] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:45.604596 2378 ts_tablet_manager.cc:1403] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.000s
I20251026 02:56:45.605692 2378 raft_consensus.cc:359] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.606046 2378 raft_consensus.cc:385] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.606166 2378 raft_consensus.cc:740] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.606576 2378 consensus_queue.cc:260] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.607788 2377 rpcz_store.cc:276] 1026 02:56:40.389797 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.389849 (+ 52us) service_pool.cc:225] Handling call
1026 02:56:45.205429 (+4815580us) ts_tablet_manager.cc:1151] Deleting tablet 8a076043f05c42d9afd9699fb4bec5ba
1026 02:56:45.205437 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.205514 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4815219}
I20251026 02:56:45.608305 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 867) took 5214 ms. Trace:
I20251026 02:56:45.608400 2377 rpcz_store.cc:276] 1026 02:56:40.394345 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.399612 (+ 5267us) service_pool.cc:225] Handling call
1026 02:56:45.608104 (+5208492us) ts_tablet_manager.cc:1151] Deleting tablet 9b9a5daa36b547a7888d38dc3329eaf2
1026 02:56:45.608118 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.608287 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5208251}
I20251026 02:56:45.608786 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 868) took 5198 ms. Trace:
I20251026 02:56:45.608870 2377 rpcz_store.cc:276] 1026 02:56:40.410641 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.410708 (+ 67us) service_pool.cc:225] Handling call
1026 02:56:45.608656 (+5197948us) ts_tablet_manager.cc:1151] Deleting tablet 87895fb88c284a4f8d7c58a5ee48ba8e
1026 02:56:45.608669 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.608770 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5197704}
I20251026 02:56:45.613689 4013 tablet.cc:1722] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:45.616879 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 869) took 5190 ms. Trace:
I20251026 02:56:45.616982 2377 rpcz_store.cc:276] 1026 02:56:40.426609 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.426666 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:45.615290 (+5188624us) ts_tablet_manager.cc:1151] Deleting tablet 7096b78402074049b9c8ed1b9fe21f0e
1026 02:56:45.616758 (+ 1468us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.616858 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5188070}
I20251026 02:56:45.617280 2377 tablet_replica.cc:333] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:45.617455 2377 raft_consensus.cc:2243] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:45.617765 2377 raft_consensus.cc:2272] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.618182 2376 ts_tablet_manager.cc:1916] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.624303 2377 ts_tablet_manager.cc:1916] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:45.627094 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ff405b3d7584fa9b587d962606e80c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:45.627247 2378 ts_tablet_manager.cc:1434] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251026 02:56:45.627736 2378 tablet_bootstrap.cc:492] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:45.628332 1647 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 3131bc2948e34c25a1bbda90f8dabeb3 (table default.Table_1d748ce60f124393b4f4ad96a131fbaf [id=65f99e4eec4044b9b9a12835401e111e], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3131bc2948e34c25a1bbda90f8dabeb3 consensus state has no leader
W20251026 02:56:45.629441 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 8a076043f05c42d9afd9699fb4bec5ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a076043f05c42d9afd9699fb4bec5ba already in progress: creating tablet
W20251026 02:56:45.629725 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 9b9a5daa36b547a7888d38dc3329eaf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b9a5daa36b547a7888d38dc3329eaf2 already in progress: creating tablet
W20251026 02:56:45.629978 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 87895fb88c284a4f8d7c58a5ee48ba8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87895fb88c284a4f8d7c58a5ee48ba8e already in progress: creating tablet
W20251026 02:56:45.631035 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7096b78402074049b9c8ed1b9fe21f0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7096b78402074049b9c8ed1b9fe21f0e already in progress: creating tablet
I20251026 02:56:45.633008 2375 tablet_bootstrap.cc:492] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:45.602757 4015 tablet.cc:1722] T 882cd1680aa140f6a63a961cd4aec25a P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:45.640818 2375 tablet_bootstrap.cc:654] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.651229 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_03ccdc56e6a14ea9af51795ee3c3fc08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:45.651851 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03ccdc56e6a14ea9af51795ee3c3fc08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:45.654119 4020 tablet.cc:1722] T ff242ef9541a42d793c4484190956da6 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251026 02:56:45.655150 2195 consensus_peers.cc:597] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6f9ae327aa4401baf66a737eaa93cf6. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:45.659823 2377 ts_tablet_manager.cc:1929] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.660988 2377 log.cc:1199] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/6a3491b303f94fe2a2c838297f73d464
I20251026 02:56:45.661362 2377 ts_tablet_manager.cc:1950] T 6a3491b303f94fe2a2c838297f73d464 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:45.661670 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 870) took 5221 ms. Trace:
I20251026 02:56:45.661749 2377 rpcz_store.cc:276] 1026 02:56:40.440335 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.440394 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:45.617231 (+5176837us) ts_tablet_manager.cc:1151] Deleting tablet 6a3491b303f94fe2a2c838297f73d464
1026 02:56:45.617242 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.652294 (+ 35052us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.659719 (+ 7425us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.661655 (+ 1936us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5869,"tablet-delete.queue_time_us":5176560}
I20251026 02:56:45.662072 2377 tablet_replica.cc:333] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:45.664500 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 6a3491b303f94fe2a2c838297f73d464 (table default.Table_30b96d2d4369458390803c6be18efe0e [id=9fd6e480fc3543149959773e691a3ddd]) successfully deleted
W20251026 02:56:45.599345 2053 leader_election.cc:343] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.621681 2372 ts_tablet_manager.cc:1929] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.666744 2372 log.cc:1199] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/bfa68f2c942c43e2a02e9d333bdfa2a2
W20251026 02:56:45.666743 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 67418.9us
I20251026 02:56:45.667085 2053 leader_election.cc:304] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:45.667220 2372 ts_tablet_manager.cc:1950] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:45.667547 2053 leader_election.cc:343] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.667865 3988 raft_consensus.cc:2804] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:45.668059 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 832) took 6077 ms. Trace:
I20251026 02:56:45.668143 2372 rpcz_store.cc:276] 1026 02:56:39.590425 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.590481 (+ 56us) service_pool.cc:225] Handling call
1026 02:56:45.563659 (+5973178us) ts_tablet_manager.cc:1151] Deleting tablet bfa68f2c942c43e2a02e9d333bdfa2a2
1026 02:56:45.563671 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.602575 (+ 38904us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.621567 (+ 18992us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.668038 (+ 46471us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5972991}
I20251026 02:56:45.668305 3988 raft_consensus.cc:697] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:45.668536 2372 tablet_replica.cc:333] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:45.668707 2372 raft_consensus.cc:2243] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:45.668977 2377 raft_consensus.cc:2243] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:45.668907 3988 consensus_queue.cc:237] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.669309 2377 raft_consensus.cc:2272] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.669977 2377 ts_tablet_manager.cc:1916] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.670429 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet bfa68f2c942c43e2a02e9d333bdfa2a2 (table default.Table_f51a5406d89743649dab37ef175a4666 [id=84eea78390b0408dbfeda88cafa9dcd9]) successfully deleted
I20251026 02:56:45.671175 2053 leader_election.cc:304] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:45.671440 2372 raft_consensus.cc:2272] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.671468 3849 raft_consensus.cc:2804] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:45.671530 3849 raft_consensus.cc:493] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:45.671583 3849 raft_consensus.cc:3060] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:45.671986 2372 ts_tablet_manager.cc:1916] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.674294 3849 raft_consensus.cc:515] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.674861 3849 leader_election.cc:290] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:45.675196 2053 leader_election.cc:343] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:45.677597 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77ecdcbe25e24267bf1066a1baad804a consensus state has no leader
W20251026 02:56:45.680038 2053 leader_election.cc:343] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.683811 2381 tablet_bootstrap.cc:492] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:45.684016 2381 ts_tablet_manager.cc:1403] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.132s user 0.009s sys 0.025s
I20251026 02:56:45.684969 2381 raft_consensus.cc:359] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.685230 2381 raft_consensus.cc:385] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.685319 2381 raft_consensus.cc:740] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.685619 2381 consensus_queue.cc:260] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.686399 2381 ts_tablet_manager.cc:1434] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:45.688938 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet fe48472efa444bf580e4074c96a9de92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.689204 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b42f8c686f234486a22954c2d04a51b5" } new_table_name: "default.Table_6b5b0a005339470aa86ae92151ee1920" modify_external_catalogs: true
I20251026 02:56:45.691215 4017 tablet.cc:1722] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:45.689273 4015 tablet.cc:1722] T ff242ef9541a42d793c4484190956da6 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:45.693282 2053 leader_election.cc:304] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:45.693513 3849 raft_consensus.cc:2749] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:45.693818 3984 raft_consensus.cc:493] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.693940 3984 raft_consensus.cc:515] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:45.694008 2053 leader_election.cc:343] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:45.694257 2053 leader_election.cc:343] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:45.694460 2053 leader_election.cc:343] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.694504 3984 leader_election.cc:290] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.694517 2053 leader_election.cc:304] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:45.694752 3849 raft_consensus.cc:2749] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:45.695077 2053 leader_election.cc:343] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.695147 2053 leader_election.cc:304] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:45.700479 1915 leader_election.cc:343] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: fe48472efa444bf580e4074c96a9de92
I20251026 02:56:45.701269 2378 tablet_bootstrap.cc:654] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:45.702782 2195 consensus_peers.cc:597] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f437d8b5f5784e8fbec10d0700e90348. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:45.709570 4021 tablet.cc:1722] T ff242ef9541a42d793c4484190956da6 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:45.712065 3849 raft_consensus.cc:2749] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:45.712332 1916 leader_election.cc:343] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: fe48472efa444bf580e4074c96a9de92
I20251026 02:56:45.712376 3849 raft_consensus.cc:493] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.712481 1916 leader_election.cc:304] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:45.712591 3849 raft_consensus.cc:515] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.712826 3984 raft_consensus.cc:2749] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:45.713238 3849 leader_election.cc:290] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.716127 2381 tablet_bootstrap.cc:492] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:45.720764 2378 tablet_bootstrap.cc:492] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:45.723551 2378 ts_tablet_manager.cc:1403] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.000s
I20251026 02:56:45.724637 2378 raft_consensus.cc:359] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.728781 2378 raft_consensus.cc:385] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.729010 2378 raft_consensus.cc:740] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.690898 2372 ts_tablet_manager.cc:1929] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.730154 2372 log.cc:1199] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3e38f282e8414b7db2a6d5f4a8933e84
I20251026 02:56:45.730583 2372 ts_tablet_manager.cc:1950] T 3e38f282e8414b7db2a6d5f4a8933e84 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:45.731411 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 833) took 6140 ms. Trace:
I20251026 02:56:45.730551 2378 consensus_queue.cc:260] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.731519 2372 rpcz_store.cc:276] 1026 02:56:39.590563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.590613 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:45.668492 (+6077879us) ts_tablet_manager.cc:1151] Deleting tablet 3e38f282e8414b7db2a6d5f4a8933e84
1026 02:56:45.668501 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.671245 (+ 2744us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x6170000c0b70. stack: 00007f8f76bbd9f6 00007f8f76bbd7b0 00007f8f75b993ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8f8296276f 00007f8f82a12f5c 00007f8f82a831e3 00007f8f83dd0ae3 00007f8f8698d287 00007f8f869c02d4 00007f8f869a0d70 00007f8f869a0afd 0000000000a8867c 00007f8f76c03d7f
1026 02:56:45.675257 (+ 4012us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.690791 (+ 15534us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.731384 (+ 40593us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4777856,"tablet-delete.queue_time_us":6077750}
I20251026 02:56:45.731966 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 835) took 6124 ms. Trace:
I20251026 02:56:45.732055 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3e38f282e8414b7db2a6d5f4a8933e84 (table default.Table_3a75d3c1b8304d1bbcc19c12111f24dd [id=e2f6dfa3686b41ddbc9f3b0205c7f05f]) successfully deleted
I20251026 02:56:45.732064 2372 rpcz_store.cc:276] 1026 02:56:39.607411 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.612276 (+ 4865us) service_pool.cc:225] Handling call
1026 02:56:45.731826 (+6119550us) ts_tablet_manager.cc:1151] Deleting tablet 112181b9207e448f959f8311ad95a02b
1026 02:56:45.731836 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.731951 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6119370}
I20251026 02:56:45.732460 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 837) took 6104 ms. Trace:
I20251026 02:56:45.732537 2372 rpcz_store.cc:276] 1026 02:56:39.627578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.627624 (+ 46us) service_pool.cc:225] Handling call
1026 02:56:45.732363 (+6104739us) ts_tablet_manager.cc:1151] Deleting tablet 7961528688cb41e5be1ba380f954f66a
1026 02:56:45.732371 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.732445 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6104560}
I20251026 02:56:45.732765 2372 tablet_replica.cc:333] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:45.732790 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 112181b9207e448f959f8311ad95a02b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 112181b9207e448f959f8311ad95a02b already in progress: creating tablet
I20251026 02:56:45.732932 2372 raft_consensus.cc:2243] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:45.733074 2372 raft_consensus.cc:2272] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:45.733343 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7961528688cb41e5be1ba380f954f66a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7961528688cb41e5be1ba380f954f66a already in progress: creating tablet
I20251026 02:56:45.733924 2372 ts_tablet_manager.cc:1916] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.736613 3893 consensus_queue.cc:1048] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:45.737475 2381 tablet_bootstrap.cc:654] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.738119 2378 ts_tablet_manager.cc:1434] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251026 02:56:45.739444 2378 tablet_bootstrap.cc:492] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:45.743404 2375 tablet_bootstrap.cc:492] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:45.743541 2375 ts_tablet_manager.cc:1403] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.111s user 0.003s sys 0.001s
I20251026 02:56:45.727109 2376 ts_tablet_manager.cc:1929] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.743754 2376 log.cc:1199] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/741792cf96164695b03de3fb86b12602
I20251026 02:56:45.744184 2375 raft_consensus.cc:359] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.744452 2375 raft_consensus.cc:385] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.744551 2375 raft_consensus.cc:740] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.744804 2375 consensus_queue.cc:260] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.745823 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ea364c4a5cf4f3f99d9ff6904add697" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:45.747398 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ea364c4a5cf4f3f99d9ff6904add697" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:45.747519 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c7efb70c0804c149f36440a4a9e6ef8" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:45.747625 2262 raft_consensus.cc:3060] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:45.747917 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c7efb70c0804c149f36440a4a9e6ef8" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:45.746520 3849 raft_consensus.cc:493] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.749846 3849 raft_consensus.cc:515] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.750204 2262 raft_consensus.cc:2468] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:45.750411 3849 leader_election.cc:290] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.746680 3997 raft_consensus.cc:493] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.750605 3997 raft_consensus.cc:515] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.751094 3997 leader_election.cc:290] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:45.746838 2053 leader_election.cc:343] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.752112 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "386e1bd4635b4a7e8e7f249b577fd978" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:45.752264 2376 ts_tablet_manager.cc:1950] T 741792cf96164695b03de3fb86b12602 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:45.753014 2381 tablet_bootstrap.cc:492] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:45.753204 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "386e1bd4635b4a7e8e7f249b577fd978" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:45.753237 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85420a102dae4269811dbdc64aebb20a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:45.754144 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85420a102dae4269811dbdc64aebb20a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:45.754453 2053 leader_election.cc:343] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.754585 2378 tablet_bootstrap.cc:654] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.754755 2053 leader_election.cc:304] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:45.755378 2375 ts_tablet_manager.cc:1434] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251026 02:56:45.753206 2381 ts_tablet_manager.cc:1403] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.001s
I20251026 02:56:45.761121 2381 raft_consensus.cc:359] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.761364 2381 raft_consensus.cc:385] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.761514 2381 raft_consensus.cc:740] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.761840 2381 consensus_queue.cc:260] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.762463 2381 ts_tablet_manager.cc:1434] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:45.764669 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 53264ef08ff7454faa19b18fa35bd803
I20251026 02:56:45.765273 1663 catalog_manager.cc:6719] ReplaceTablet: tablet 53264ef08ff7454faa19b18fa35bd803 deleted and replaced by tablet dbacfecda16c47abad70b219ca3e5a24
I20251026 02:56:45.770130 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 53264ef08ff7454faa19b18fa35bd803 with delete_type TABLET_DATA_DELETED (replaced by tablet dbacfecda16c47abad70b219ca3e5a24) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:45.771302 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f840050f7e314d74ac6e604354a7d4c9 consensus state has no leader
I20251026 02:56:45.773263 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 885c10070d6942f7abe52c4558d887e6
I20251026 02:56:45.773738 1659 catalog_manager.cc:6719] ReplaceTablet: tablet 885c10070d6942f7abe52c4558d887e6 deleted and replaced by tablet 73f0b1443e2c4294a222e62433f3b435
I20251026 02:56:45.774675 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 885c10070d6942f7abe52c4558d887e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 73f0b1443e2c4294a222e62433f3b435) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.776882 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fef50bc3bf28484f9b1c67ab84027952" } modify_external_catalogs: true
I20251026 02:56:45.755554 3997 raft_consensus.cc:2749] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:45.777055 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fef50bc3bf28484f9b1c67ab84027952" } modify_external_catalogs: true
I20251026 02:56:45.766690 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 53264ef08ff7454faa19b18fa35bd803 with delete_type TABLET_DATA_DELETED (replaced by tablet dbacfecda16c47abad70b219ca3e5a24) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:45.777566 2375 tablet_bootstrap.cc:492] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:45.779071 2375 tablet_bootstrap.cc:654] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.779201 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3629bba6ed4d45a292b6c65c325752ec" } modify_external_catalogs: true
I20251026 02:56:45.779400 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3629bba6ed4d45a292b6c65c325752ec" } modify_external_catalogs: true
I20251026 02:56:45.781172 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet aa10cd990a944eab8d9eb66aaf20080c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.781592 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f83eb680ec4a4220a61c02921eac359a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:45.782141 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f83eb680ec4a4220a61c02921eac359a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:45.783689 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3f4de82346384f0faf6482156a0457d1" } new_table_name: "default.Table_2cb9ec57edc14123bc59106065c35a67" modify_external_catalogs: true
I20251026 02:56:45.784543 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f3f28520bd3a44738d0e86f989470e12" } modify_external_catalogs: true
I20251026 02:56:45.784729 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f3f28520bd3a44738d0e86f989470e12" } modify_external_catalogs: true
I20251026 02:56:45.785408 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 53264ef08ff7454faa19b18fa35bd803 with delete_type TABLET_DATA_DELETED (replaced by tablet dbacfecda16c47abad70b219ca3e5a24) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:45.785699 2225 tablet_service.cc:1552] Processing DeleteTablet for tablet 885c10070d6942f7abe52c4558d887e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 73f0b1443e2c4294a222e62433f3b435) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:45.786190 2194 consensus_peers.cc:597] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:45.775517 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 885c10070d6942f7abe52c4558d887e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 73f0b1443e2c4294a222e62433f3b435) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:45.773663 2053 leader_election.cc:304] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:45.788233 3997 raft_consensus.cc:2804] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:45.788355 3997 raft_consensus.cc:697] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:45.788575 3997 consensus_queue.cc:237] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.791498 2377 ts_tablet_manager.cc:1929] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:45.791644 2377 log.cc:1199] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/340bf8c78a88413299b61a52f0a578f3
I20251026 02:56:45.792025 2377 ts_tablet_manager.cc:1950] T 340bf8c78a88413299b61a52f0a578f3 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:45.796504 2381 tablet_bootstrap.cc:492] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:45.797087 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 822) took 5388 ms. Trace:
I20251026 02:56:45.797183 2376 rpcz_store.cc:276] 1026 02:56:40.408327 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.408396 (+ 69us) service_pool.cc:225] Handling call
1026 02:56:45.580013 (+5171617us) ts_tablet_manager.cc:1151] Deleting tablet 741792cf96164695b03de3fb86b12602
1026 02:56:45.580149 (+ 136us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.621566 (+ 41417us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.726994 (+105428us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.797067 (+ 70073us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":768,"tablet-delete.queue_time_us":5163507}
I20251026 02:56:45.797683 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 741792cf96164695b03de3fb86b12602 (table default.Table_d4304259ac684b8097cdbf5511ab67d7 [id=d64901838ed94621ad11692d9e0aa80e]) successfully deleted
I20251026 02:56:45.798157 2376 tablet_replica.cc:333] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:45.798174 2381 tablet_bootstrap.cc:654] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.798388 2376 raft_consensus.cc:2243] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:45.798568 2376 raft_consensus.cc:2272] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.799157 2376 ts_tablet_manager.cc:1916] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.804028 4013 tablet.cc:1722] T 23062e7292f44ad5ad9dfc22a401741c P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:45.805030 4015 tablet.cc:1722] T 23062e7292f44ad5ad9dfc22a401741c P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251026 02:56:45.806116 2053 leader_election.cc:343] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:45.806289 2053 leader_election.cc:343] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:45.806402 2053 leader_election.cc:343] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:45.806622 2053 leader_election.cc:343] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.806684 2053 leader_election.cc:304] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:45.808475 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 872) took 5129 ms. Trace:
I20251026 02:56:45.808565 2377 rpcz_store.cc:276] 1026 02:56:40.678680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.678734 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:45.662030 (+4983296us) ts_tablet_manager.cc:1151] Deleting tablet 340bf8c78a88413299b61a52f0a578f3
1026 02:56:45.662039 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.718056 (+ 56017us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.791363 (+ 73307us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.808458 (+ 17095us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4618,"tablet-delete.queue_time_us":4980523}
I20251026 02:56:45.808729 3849 raft_consensus.cc:2749] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:45.808964 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 879) took 4881 ms. Trace:
I20251026 02:56:45.809026 2377 rpcz_store.cc:276] 1026 02:56:40.927493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.928731 (+ 1238us) service_pool.cc:225] Handling call
1026 02:56:45.808856 (+4880125us) ts_tablet_manager.cc:1151] Deleting tablet 572afa86823247a8b1744813350c8eff
1026 02:56:45.808865 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.808950 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4879925}
I20251026 02:56:45.809244 2377 tablet_replica.cc:333] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539: stopping tablet replica
W20251026 02:56:45.809340 2053 leader_election.cc:343] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:45.809434 2053 leader_election.cc:304] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:45.809561 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 340bf8c78a88413299b61a52f0a578f3 (table default.Table_26dd741c83ff419f8c9bc808e83d9c72 [id=c162ef89d2f14e91a24a6d8c2c37c261]) successfully deleted
W20251026 02:56:45.809984 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 572afa86823247a8b1744813350c8eff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 572afa86823247a8b1744813350c8eff already in progress: creating tablet
I20251026 02:56:45.810473 2377 raft_consensus.cc:2243] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 2 LEADER]: Raft consensus shutting down.
I20251026 02:56:45.810734 2377 pending_rounds.cc:70] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:45.810782 2377 pending_rounds.cc:77] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7214888569662332928 op_type: NO_OP noop_request { }
I20251026 02:56:45.810873 3988 raft_consensus.cc:2749] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:45.810895 2377 raft_consensus.cc:2889] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:45.810969 2377 raft_consensus.cc:2272] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.811496 2377 ts_tablet_manager.cc:1916] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.812634 2376 ts_tablet_manager.cc:1929] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:45.815894 4001 raft_consensus.cc:493] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 209968db4dee47c49497a66c3ee750b7)
I20251026 02:56:45.816166 4001 raft_consensus.cc:515] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:45.817114 2376 log.cc:1199] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/be1c0948c0ae4563b7cb644521914cb9
I20251026 02:56:45.818251 2376 ts_tablet_manager.cc:1950] T be1c0948c0ae4563b7cb644521914cb9 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:45.820887 3957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209968db4dee47c49497a66c3ee750b7. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:45.821136 3957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 02:56:45.822748 4001 leader_election.cc:290] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 209968db4dee47c49497a66c3ee750b7 (127.1.150.62:44525), 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655)
I20251026 02:56:45.827913 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 823) took 5418 ms. Trace:
I20251026 02:56:45.828622 2376 rpcz_store.cc:276] 1026 02:56:40.409020 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.409108 (+ 88us) service_pool.cc:225] Handling call
1026 02:56:45.798101 (+5388993us) ts_tablet_manager.cc:1151] Deleting tablet be1c0948c0ae4563b7cb644521914cb9
1026 02:56:45.798112 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.802717 (+ 4605us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.812540 (+ 9823us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.827890 (+ 15350us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5381529}
I20251026 02:56:45.829490 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet be1c0948c0ae4563b7cb644521914cb9 (table default.Table_d197f56404544c8cb996d53edb9559a4 [id=777761bfdc67409bb98be7cf82bfa547]) successfully deleted
I20251026 02:56:45.830117 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 824) took 5420 ms. Trace:
I20251026 02:56:45.830614 2376 rpcz_store.cc:276] 1026 02:56:40.409878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.409952 (+ 74us) service_pool.cc:225] Handling call
1026 02:56:45.829969 (+5420017us) ts_tablet_manager.cc:1151] Deleting tablet 87895fb88c284a4f8d7c58a5ee48ba8e
1026 02:56:45.829985 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.830101 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5413344}
W20251026 02:56:45.831282 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 87895fb88c284a4f8d7c58a5ee48ba8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87895fb88c284a4f8d7c58a5ee48ba8e already in progress: creating tablet
I20251026 02:56:45.831722 2376 tablet_replica.cc:333] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:45.832154 2376 raft_consensus.cc:2243] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:45.832630 2376 raft_consensus.cc:2272] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.833331 2376 ts_tablet_manager.cc:1916] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.842849 3984 raft_consensus.cc:493] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:45.843055 3984 raft_consensus.cc:515] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.843832 3984 leader_election.cc:290] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.845221 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d212170f056a44dfb15cfcb7f9fbae2f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:45.845417 1916 leader_election.cc:343] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: d212170f056a44dfb15cfcb7f9fbae2f
I20251026 02:56:45.845472 2259 raft_consensus.cc:2468] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:45.846609 1915 leader_election.cc:304] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:45.847513 3984 raft_consensus.cc:2804] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:45.847592 3984 raft_consensus.cc:493] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:45.847644 3984 raft_consensus.cc:3060] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:45.850412 3984 raft_consensus.cc:515] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.851730 3984 leader_election.cc:290] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:45.852025 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d212170f056a44dfb15cfcb7f9fbae2f" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:45.852253 2259 raft_consensus.cc:3060] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:45.852540 1916 leader_election.cc:343] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: d212170f056a44dfb15cfcb7f9fbae2f
I20251026 02:56:45.855365 2259 raft_consensus.cc:2468] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:45.856104 1915 leader_election.cc:304] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:45.856638 3984 raft_consensus.cc:2804] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:45.861652 3984 raft_consensus.cc:697] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:45.862032 3984 consensus_queue.cc:237] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.866413 2376 ts_tablet_manager.cc:1929] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:45.866619 2376 log.cc:1199] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/340bf8c78a88413299b61a52f0a578f3
I20251026 02:56:45.867041 2376 ts_tablet_manager.cc:1950] T 340bf8c78a88413299b61a52f0a578f3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:45.869539 4040 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20251026 02:56:45.870903 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f383f638e63040379bf749b8774fd7ed consensus state has no leader
I20251026 02:56:45.873996 2372 ts_tablet_manager.cc:1929] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:45.874136 2372 log.cc:1199] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/c2d6691bb4cf4a8292b0c082e54a91a5
I20251026 02:56:45.874723 2372 ts_tablet_manager.cc:1950] T c2d6691bb4cf4a8292b0c082e54a91a5 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:45.874845 2381 tablet_bootstrap.cc:492] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:45.875033 2381 ts_tablet_manager.cc:1403] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.002s
I20251026 02:56:45.875864 2381 raft_consensus.cc:359] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.876173 2381 raft_consensus.cc:385] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.876303 2381 raft_consensus.cc:740] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.876641 2381 consensus_queue.cc:260] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.877194 2381 ts_tablet_manager.cc:1434] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:45.877529 2381 tablet_bootstrap.cc:492] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:45.879169 2381 tablet_bootstrap.cc:654] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.894766 1978 raft_consensus.cc:3060] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:45.897221 1744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 4 candidate_status { last_received { term: 3 index: 937 } } ignore_live_leader: false dest_uuid: "647c0de4ce394caca75179fe11658fc8" is_pre_election: true
I20251026 02:56:45.909053 2375 tablet_bootstrap.cc:492] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:45.909363 2375 ts_tablet_manager.cc:1403] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.132s user 0.003s sys 0.001s
I20251026 02:56:45.910399 2375 raft_consensus.cc:359] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.910719 2375 raft_consensus.cc:385] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.910835 2375 raft_consensus.cc:740] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.911214 2375 consensus_queue.cc:260] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.911943 2375 ts_tablet_manager.cc:1434] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:45.912859 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 826) took 5234 ms. Trace:
I20251026 02:56:45.912959 2376 rpcz_store.cc:276] 1026 02:56:40.677937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.678023 (+ 86us) service_pool.cc:225] Handling call
1026 02:56:45.831676 (+5153653us) ts_tablet_manager.cc:1151] Deleting tablet 340bf8c78a88413299b61a52f0a578f3
1026 02:56:45.831686 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.836962 (+ 5276us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.866300 (+ 29338us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.912829 (+ 46529us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5153463}
I20251026 02:56:45.913386 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 833) took 5060 ms. Trace:
I20251026 02:56:45.913473 2376 rpcz_store.cc:276] 1026 02:56:40.852943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.856305 (+ 3362us) service_pool.cc:225] Handling call
1026 02:56:45.913244 (+5056939us) ts_tablet_manager.cc:1151] Deleting tablet 572afa86823247a8b1744813350c8eff
1026 02:56:45.913254 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.913369 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5056731}
I20251026 02:56:45.913794 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 834) took 5046 ms. Trace:
I20251026 02:56:45.913858 2376 rpcz_store.cc:276] 1026 02:56:40.866860 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.866908 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:45.913670 (+5046762us) ts_tablet_manager.cc:1151] Deleting tablet 7c148c74fbcd4d30b9ba0ea059284427
1026 02:56:45.913699 (+ 29us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.913778 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5046570}
I20251026 02:56:45.914120 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 835) took 5037 ms. Trace:
I20251026 02:56:45.914181 2376 rpcz_store.cc:276] 1026 02:56:40.877076 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.877488 (+ 412us) service_pool.cc:225] Handling call
1026 02:56:45.914032 (+5036544us) ts_tablet_manager.cc:1151] Deleting tablet 83416598159a4eb598cf91a4a4e6f44a
1026 02:56:45.914040 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.914106 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5036256}
I20251026 02:56:45.914441 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 836) took 5001 ms. Trace:
I20251026 02:56:45.914499 2376 rpcz_store.cc:276] 1026 02:56:40.912526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.912587 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:45.914354 (+5001767us) ts_tablet_manager.cc:1151] Deleting tablet 9d12453c4225455881336b9f31d8e378
1026 02:56:45.914361 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.914420 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5001554}
I20251026 02:56:45.914718 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 838) took 4933 ms. Trace:
I20251026 02:56:45.914768 2376 rpcz_store.cc:276] 1026 02:56:40.980841 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.980893 (+ 52us) service_pool.cc:225] Handling call
1026 02:56:45.914649 (+4933756us) ts_tablet_manager.cc:1151] Deleting tablet 7f22c6c163c34ed48a0b7eeb5a574e76
1026 02:56:45.914655 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.914708 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4933580}
I20251026 02:56:45.914986 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 840) took 4902 ms. Trace:
I20251026 02:56:45.915035 2376 rpcz_store.cc:276] 1026 02:56:41.012267 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.012349 (+ 82us) service_pool.cc:225] Handling call
1026 02:56:45.914919 (+4902570us) ts_tablet_manager.cc:1151] Deleting tablet a32b1a6b856c47b993a308db94dac357
1026 02:56:45.914925 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.914976 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4902379}
I20251026 02:56:45.915220 2376 tablet_replica.cc:333] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:45.915205 2381 tablet_bootstrap.cc:492] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:45.915354 2381 ts_tablet_manager.cc:1403] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20251026 02:56:45.915385 2376 raft_consensus.cc:2243] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:45.915555 2376 raft_consensus.cc:2272] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.916121 2376 ts_tablet_manager.cc:1916] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:45.916118 2381 raft_consensus.cc:359] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.916333 2381 raft_consensus.cc:385] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.916391 2381 raft_consensus.cc:740] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.916663 2381 consensus_queue.cc:260] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.917292 2381 ts_tablet_manager.cc:1434] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:45.917578 2381 tablet_bootstrap.cc:492] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:45.919358 2381 tablet_bootstrap.cc:654] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.920588 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 340bf8c78a88413299b61a52f0a578f3 (table default.Table_26dd741c83ff419f8c9bc808e83d9c72 [id=c162ef89d2f14e91a24a6d8c2c37c261]) successfully deleted
I20251026 02:56:45.920696 3362 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" instance_seqno: 1761447379602867) as {username='slave'} at 127.1.150.2:51761; Asking this server to re-register.
W20251026 02:56:45.920939 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 572afa86823247a8b1744813350c8eff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 572afa86823247a8b1744813350c8eff already in progress: creating tablet
W20251026 02:56:45.921160 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7c148c74fbcd4d30b9ba0ea059284427 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c148c74fbcd4d30b9ba0ea059284427 already in progress: creating tablet
W20251026 02:56:45.921340 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 83416598159a4eb598cf91a4a4e6f44a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83416598159a4eb598cf91a4a4e6f44a already in progress: creating tablet
W20251026 02:56:45.921543 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 9d12453c4225455881336b9f31d8e378 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d12453c4225455881336b9f31d8e378 already in progress: creating tablet
W20251026 02:56:45.921726 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7f22c6c163c34ed48a0b7eeb5a574e76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f22c6c163c34ed48a0b7eeb5a574e76 already in progress: creating tablet
W20251026 02:56:45.921964 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet a32b1a6b856c47b993a308db94dac357 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a32b1a6b856c47b993a308db94dac357 already in progress: creating tablet
I20251026 02:56:45.925928 1673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 4 candidate_status { last_received { term: 3 index: 937 } } ignore_live_leader: false dest_uuid: "209968db4dee47c49497a66c3ee750b7" is_pre_election: true
I20251026 02:56:45.926697 3327 leader_election.cc:304] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 751b719b75ce4bc2a88a19091230ecb7; no voters: 209968db4dee47c49497a66c3ee750b7, 647c0de4ce394caca75179fe11658fc8
I20251026 02:56:45.927597 4001 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 02:56:45.930721 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9b976c87b5d04f9294add7761a2afbfd
I20251026 02:56:45.932195 1657 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet dbacfecda16c47abad70b219ca3e5a24
I20251026 02:56:45.933461 3391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57034 (request call id 0) took 7882 ms. Trace:
I20251026 02:56:45.938714 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b976c87b5d04f9294add7761a2afbfd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:45 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:45.939105 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b976c87b5d04f9294add7761a2afbfd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:45 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:45.953260 2381 tablet_bootstrap.cc:492] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:45.953576 2381 ts_tablet_manager.cc:1403] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20251026 02:56:45.954447 2381 raft_consensus.cc:359] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.954699 2381 raft_consensus.cc:385] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.954803 2381 raft_consensus.cc:740] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.955104 2381 consensus_queue.cc:260] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.955256 2376 ts_tablet_manager.cc:1929] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251026 02:56:45.955925 2381 ts_tablet_manager.cc:1434] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:45.956308 2381 tablet_bootstrap.cc:492] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:45.957909 2381 tablet_bootstrap.cc:654] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:45.958288 3362 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" instance_seqno: 1761447379108470) as {username='slave'} at 127.1.150.1:42831; Asking this server to re-register.
I20251026 02:56:45.959249 2376 log.cc:1199] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/dbf2a1ce78154415b23ab6d0efbc1f52
I20251026 02:56:45.959692 2376 ts_tablet_manager.cc:1950] T dbf2a1ce78154415b23ab6d0efbc1f52 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:45.961628 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 844) took 4861 ms. Trace:
I20251026 02:56:45.961755 2376 rpcz_store.cc:276] 1026 02:56:41.100061 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.100120 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:45.915182 (+4815062us) ts_tablet_manager.cc:1151] Deleting tablet dbf2a1ce78154415b23ab6d0efbc1f52
1026 02:56:45.915189 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.927648 (+ 12459us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.955145 (+ 27497us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.961611 (+ 6466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4814840}
I20251026 02:56:45.962327 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet dbf2a1ce78154415b23ab6d0efbc1f52 (table default.Table_22ae79375e8e49d4b6d146d087ffa64b [id=4c170eaba0fd422f9fb0ce5e63e5c746]) successfully deleted
I20251026 02:56:45.962795 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 848) took 4692 ms. Trace:
I20251026 02:56:45.962914 2376 rpcz_store.cc:276] 1026 02:56:41.269934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.270018 (+ 84us) service_pool.cc:225] Handling call
1026 02:56:45.962663 (+4692645us) ts_tablet_manager.cc:1151] Deleting tablet 94bdaa97e7604040b7fc94fe22e743bf
1026 02:56:45.962673 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.962777 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4692444}
W20251026 02:56:45.963958 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 94bdaa97e7604040b7fc94fe22e743bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94bdaa97e7604040b7fc94fe22e743bf already in progress: creating tablet
I20251026 02:56:45.961177 2027 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:45.964916 2376 tablet_replica.cc:333] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:45.965487 2376 raft_consensus.cc:2243] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:45.968596 2376 raft_consensus.cc:2272] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:45.970575 2376 ts_tablet_manager.cc:1916] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:45.975389 2194 consensus_peers.cc:597] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:45.981015 3362 ts_manager.cc:194] Registered new tserver with Master: 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563)
I20251026 02:56:45.989033 2381 tablet_bootstrap.cc:492] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:45.989346 2381 ts_tablet_manager.cc:1403] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20251026 02:56:45.990252 2381 raft_consensus.cc:359] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.990525 2381 raft_consensus.cc:385] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:45.990633 2381 raft_consensus.cc:740] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:45.990942 2381 consensus_queue.cc:260] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:45.997087 2381 ts_tablet_manager.cc:1434] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251026 02:56:45.997610 2376 ts_tablet_manager.cc:1929] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:45.997797 2376 log.cc:1199] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/352a284902c94f088e2e6ae006162db0
I20251026 02:56:45.998291 2376 ts_tablet_manager.cc:1950] T 352a284902c94f088e2e6ae006162db0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:45.998678 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 849) took 4728 ms. Trace:
I20251026 02:56:45.998812 2376 rpcz_store.cc:276] 1026 02:56:41.270098 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.270256 (+ 158us) service_pool.cc:225] Handling call
1026 02:56:45.964863 (+4694607us) ts_tablet_manager.cc:1151] Deleting tablet 352a284902c94f088e2e6ae006162db0
1026 02:56:45.964874 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.986127 (+ 21253us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.997509 (+ 11382us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.998659 (+ 1150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4694454}
I20251026 02:56:45.999418 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 352a284902c94f088e2e6ae006162db0 (table default.Table_6cd4ee82e9ec406f971669b7f0722677 [id=414375003cd44b5c8c9f0249b339c28a]) successfully deleted
I20251026 02:56:45.999915 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 857) took 4512 ms. Trace:
I20251026 02:56:46.000036 2376 rpcz_store.cc:276] 1026 02:56:41.487921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.487981 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:45.999790 (+4511809us) ts_tablet_manager.cc:1151] Deleting tablet 3aed4584b49a4eccac2ba44454866d49
1026 02:56:45.999801 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.999897 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4511600}
W20251026 02:56:46.000593 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3aed4584b49a4eccac2ba44454866d49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3aed4584b49a4eccac2ba44454866d49 already in progress: creating tablet
I20251026 02:56:46.000972 2376 tablet_replica.cc:333] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:46.001187 2376 raft_consensus.cc:2243] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.001485 2376 raft_consensus.cc:2272] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.002099 2376 ts_tablet_manager.cc:1916] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.002566 2381 tablet_bootstrap.cc:492] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:45.922966 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 836) took 6295 ms. Trace:
I20251026 02:56:46.005596 2372 rpcz_store.cc:276] 1026 02:56:39.627479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.635384 (+ 7905us) service_pool.cc:225] Handling call
1026 02:56:45.732725 (+6097341us) ts_tablet_manager.cc:1151] Deleting tablet c2d6691bb4cf4a8292b0c082e54a91a5
1026 02:56:45.732732 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.807552 (+ 74820us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.873876 (+ 66324us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:45.922936 (+ 49060us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2432,"tablet-delete.queue_time_us":6097145}
I20251026 02:56:46.005859 2375 tablet_bootstrap.cc:492] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:46.006068 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 841) took 6329 ms. Trace:
I20251026 02:56:46.006136 2372 rpcz_store.cc:276] 1026 02:56:39.676921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.676975 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:46.005928 (+6328953us) ts_tablet_manager.cc:1151] Deleting tablet ec94273925e94fb09f55249426bc4eaa
1026 02:56:46.005938 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.006053 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6295384}
I20251026 02:56:45.924567 2165 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:45.937225 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b976c87b5d04f9294add7761a2afbfd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:45 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.006729 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 840) took 6335 ms. Trace:
I20251026 02:56:46.006803 2372 rpcz_store.cc:276] 1026 02:56:39.671012 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.689506 (+ 18494us) service_pool.cc:225] Handling call
1026 02:56:46.006352 (+6316846us) ts_tablet_manager.cc:1151] Deleting tablet 144b957c69e64085a6f78050dea426b4
1026 02:56:46.006618 (+ 266us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.006713 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6295425}
I20251026 02:56:46.007138 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet c2d6691bb4cf4a8292b0c082e54a91a5 (table default.Table_8602d75889ea4576ae575b55fe95e940 [id=b0104213af32435589ae19109b7aee7f]) successfully deleted
I20251026 02:56:46.007414 2375 tablet_bootstrap.cc:654] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:46.007427 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ec94273925e94fb09f55249426bc4eaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec94273925e94fb09f55249426bc4eaa already in progress: creating tablet
W20251026 02:56:46.008237 2195 consensus_peers.cc:597] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa10cd990a944eab8d9eb66aaf20080c. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:46.008904 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 144b957c69e64085a6f78050dea426b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 144b957c69e64085a6f78050dea426b4 already in progress: creating tablet
I20251026 02:56:46.010960 2377 ts_tablet_manager.cc:1929] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251026 02:56:46.011078 2377 log.cc:1199] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/a6f9ae327aa4401baf66a737eaa93cf6
I20251026 02:56:46.011094 3362 ts_manager.cc:194] Registered new tserver with Master: 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.013312 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 843) took 6283 ms. Trace:
I20251026 02:56:46.013430 2372 rpcz_store.cc:276] 1026 02:56:39.729813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:39.729880 (+ 67us) service_pool.cc:225] Handling call
1026 02:56:46.007014 (+6277134us) ts_tablet_manager.cc:1151] Deleting tablet b1f6e2c58c7f4367813eb66ebda5e0ca
1026 02:56:46.009680 (+ 2666us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.013290 (+ 3610us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6276915}
I20251026 02:56:46.013916 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 865) took 5725 ms. Trace:
W20251026 02:56:46.014032 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet b1f6e2c58c7f4367813eb66ebda5e0ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1f6e2c58c7f4367813eb66ebda5e0ca already in progress: creating tablet
I20251026 02:56:46.013993 2372 rpcz_store.cc:276] 1026 02:56:40.288828 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.288981 (+ 153us) service_pool.cc:225] Handling call
1026 02:56:46.013735 (+5724754us) ts_tablet_manager.cc:1151] Deleting tablet d9391e7654c442a5bb312f0daaf1d99b
1026 02:56:46.013745 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.013839 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5724564}
I20251026 02:56:46.014426 2372 tablet_replica.cc:333] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:46.014401 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_141d324fb96a4df0b7916345b94ac27d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:46.014611 2372 raft_consensus.cc:2243] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.014756 2372 raft_consensus.cc:2272] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:46.014986 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_141d324fb96a4df0b7916345b94ac27d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.015304 2372 ts_tablet_manager.cc:1916] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.016263 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b0315e3c5d3f481b9db0d3570ad5a9c2" } modify_external_catalogs: true
I20251026 02:56:46.016455 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b0315e3c5d3f481b9db0d3570ad5a9c2" } modify_external_catalogs: true
I20251026 02:56:46.017390 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_628ae32e1bb94616953205cb1a976e99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.017890 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_628ae32e1bb94616953205cb1a976e99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:46.024139 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 63eac91f5cb540e6b999ac816ee0b89d consensus state has no leader
I20251026 02:56:46.024405 2381 tablet_bootstrap.cc:654] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:46.024523 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet e4c149e8b7f9420c8dba56da2e21f3f9 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:46.024838 1656 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_7125b173069d4066a0b65f414c55c7e1 [id=159df78be6ba48a594ff75c61451cf11] alter complete (version 1)
I20251026 02:56:46.028249 3362 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "696476265495492790739ec126a08539" instance_seqno: 1761447380121209) as {username='slave'} at 127.1.150.3:53887; Asking this server to re-register.
I20251026 02:56:46.032202 2308 heartbeater.cc:461] Registering TS with master...
I20251026 02:56:46.035440 2381 tablet_bootstrap.cc:492] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:46.035668 2381 ts_tablet_manager.cc:1403] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.001s
I20251026 02:56:46.036549 3362 ts_manager.cc:194] Registered new tserver with Master: 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.036568 2381 raft_consensus.cc:359] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.036857 2381 raft_consensus.cc:385] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.036957 2381 raft_consensus.cc:740] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.037241 2381 consensus_queue.cc:260] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.043313 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 4db275efdd344817b11a2168675cf95e (table default.Table_c30cfa52b6524323bda478613d74178b [id=1fd74186ea10498599435dccd7330d25]). Expected version 1 got 0
I20251026 02:56:46.045639 2381 ts_tablet_manager.cc:1434] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251026 02:56:46.048903 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a3491b303f94fe2a2c838297f73d464 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b5e61afe90e4bc4a2897de21e0b3919) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.049115 2225 tablet_service.cc:1552] Processing DeleteTablet for tablet 62d30d79a6e4432a8dbe40a104e87fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.051757 1662 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 85420a102dae4269811dbdc64aebb20a (table default.Table_9fef71477b964049a34fada596af80bf [id=240bb6da01cf4c1b98f9301e17cc7817]). Expected version 1 got 0
I20251026 02:56:46.051995 1662 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 23062e7292f44ad5ad9dfc22a401741c (table default.Table_2cf16843709c4e2a9c3632c25c6bbe25 [id=2e17bae10e0b48a492f5ee0bed7d00d0]). Expected version 2 got 0
I20251026 02:56:46.054112 2381 tablet_bootstrap.cc:492] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:46.055491 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ea364c4a5cf4f3f99d9ff6904add697 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.061024 2381 tablet_bootstrap.cc:654] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:46.062454 2375 tablet_bootstrap.cc:492] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:46.064265 2378 tablet_bootstrap.cc:492] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:46.064333 2375 ts_tablet_manager.cc:1403] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.001s
I20251026 02:56:46.064479 2378 ts_tablet_manager.cc:1403] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.325s user 0.016s sys 0.007s
I20251026 02:56:46.065169 2378 raft_consensus.cc:359] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.065236 2375 raft_consensus.cc:359] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.065426 2378 raft_consensus.cc:385] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.065537 2378 raft_consensus.cc:740] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.065795 2378 consensus_queue.cc:260] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.066337 2378 ts_tablet_manager.cc:1434] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 02:56:46.068346 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1697506af603436e82df1f54f1c42616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:46.069051 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_810dcd19177646f483aeef93df04b9ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:46.069841 2375 raft_consensus.cc:385] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.069969 2375 raft_consensus.cc:740] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.070288 2375 consensus_queue.cc:260] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.075922 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (table default.Table_30611e161c274e8b99bd1251e5e527aa [id=b5d91d3e8a2643c09fee85976972391a]). Expected version 1 got 0
I20251026 02:56:46.076256 1655 catalog_manager.cc:5649] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:46.076663 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 4db275efdd344817b11a2168675cf95e (table default.Table_c30cfa52b6524323bda478613d74178b [id=1fd74186ea10498599435dccd7330d25]). Expected version 1 got 0
I20251026 02:56:46.076960 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet cfac07917ee34e6dba3fac5e081f6b43 (table default.Table_907991c2b0034c118065ead079f273f9 [id=a8d490f3dbc948d38c54d14e178b736e]). Expected version 1 got 0
I20251026 02:56:46.077646 2375 ts_tablet_manager.cc:1434] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
W20251026 02:56:46.077791 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_810dcd19177646f483aeef93df04b9ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:46.079013 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1697506af603436e82df1f54f1c42616 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.081183 3893 raft_consensus.cc:493] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.081355 3893 raft_consensus.cc:515] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.081960 3893 leader_election.cc:290] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.083153 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62d30d79a6e4432a8dbe40a104e87fa0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.085371 3984 raft_consensus.cc:493] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.085508 3984 raft_consensus.cc:515] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.086099 3984 leader_election.cc:290] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:46.086961 2195 leader_election.cc:343] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 62d30d79a6e4432a8dbe40a104e87fa0
I20251026 02:56:46.087409 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e38f282e8414b7db2a6d5f4a8933e84" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:46.087944 2378 tablet_bootstrap.cc:492] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:46.088089 2194 leader_election.cc:343] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.088229 2194 leader_election.cc:304] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:46.088819 1916 leader_election.cc:343] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 3e38f282e8414b7db2a6d5f4a8933e84
I20251026 02:56:46.089326 2378 tablet_bootstrap.cc:654] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:46.089938 4009 raft_consensus.cc:2749] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:46.090090 4009 raft_consensus.cc:493] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.090206 4009 raft_consensus.cc:515] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.090775 4009 leader_election.cc:290] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.092497 2377 ts_tablet_manager.cc:1950] T a6f9ae327aa4401baf66a737eaa93cf6 P 696476265495492790739ec126a08539: Deleting consensus metadata
W20251026 02:56:46.092644 2195 leader_election.cc:343] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 0cf76ef8f6af445eb0ce15c8d79136c5
I20251026 02:56:46.093068 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 2337224b44bf4bcc9062d4f8940269ab (DEFAULT_TABLE table=default.Table_a86131ef9a344c92952a42fed41f843e [id=8028f045ad6043608a9cddd9e77ea897]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.093298 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 882) took 5165 ms. Trace:
I20251026 02:56:46.093463 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2337224b44bf4bcc9062d4f8940269ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.093375 2377 rpcz_store.cc:276] 1026 02:56:40.927818 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.929108 (+ 1290us) service_pool.cc:225] Handling call
1026 02:56:45.809204 (+4880096us) ts_tablet_manager.cc:1151] Deleting tablet a6f9ae327aa4401baf66a737eaa93cf6
1026 02:56:45.809212 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:45.887773 (+ 78561us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.010846 (+123073us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.093280 (+ 82434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4879980}
I20251026 02:56:46.093735 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 880) took 5166 ms. Trace:
I20251026 02:56:46.093811 2377 rpcz_store.cc:276] 1026 02:56:40.927624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.928968 (+ 1344us) service_pool.cc:225] Handling call
1026 02:56:46.093632 (+5164664us) ts_tablet_manager.cc:1151] Deleting tablet 7c148c74fbcd4d30b9ba0ea059284427
1026 02:56:46.093642 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.093722 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5160681}
I20251026 02:56:46.094089 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 881) took 5166 ms. Trace:
I20251026 02:56:46.094146 2377 rpcz_store.cc:276] 1026 02:56:40.927729 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.929084 (+ 1355us) service_pool.cc:225] Handling call
1026 02:56:46.093994 (+5164910us) ts_tablet_manager.cc:1151] Deleting tablet 83416598159a4eb598cf91a4a4e6f44a
1026 02:56:46.094002 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.094074 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5152625}
I20251026 02:56:46.094461 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 883) took 5166 ms. Trace:
I20251026 02:56:46.094521 2377 rpcz_store.cc:276] 1026 02:56:40.927973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.928403 (+ 430us) service_pool.cc:225] Handling call
1026 02:56:46.094359 (+5165956us) ts_tablet_manager.cc:1151] Deleting tablet 9d12453c4225455881336b9f31d8e378
1026 02:56:46.094367 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.094448 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5152827}
I20251026 02:56:46.094769 2377 tablet_replica.cc:333] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:46.094935 2377 raft_consensus.cc:2243] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.095069 2377 raft_consensus.cc:2272] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.089730 1915 leader_election.cc:304] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:46.096999 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 2337224b44bf4bcc9062d4f8940269ab (DEFAULT_TABLE table=default.Table_a86131ef9a344c92952a42fed41f843e [id=8028f045ad6043608a9cddd9e77ea897]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.097397 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2337224b44bf4bcc9062d4f8940269ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.092244 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cf76ef8f6af445eb0ce15c8d79136c5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:46.099339 2194 leader_election.cc:343] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.099454 2194 leader_election.cc:304] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.099836 3893 raft_consensus.cc:2749] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:46.100034 3984 raft_consensus.cc:2749] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:46.100836 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet a6f9ae327aa4401baf66a737eaa93cf6 (table default.Table_828c16a96a47489f835a891f34d707b1 [id=432234d273ca4c9ea53c7a078c274637]) successfully deleted
W20251026 02:56:46.101176 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7c148c74fbcd4d30b9ba0ea059284427 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c148c74fbcd4d30b9ba0ea059284427 already in progress: creating tablet
W20251026 02:56:46.101405 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 83416598159a4eb598cf91a4a4e6f44a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83416598159a4eb598cf91a4a4e6f44a already in progress: creating tablet
W20251026 02:56:46.101611 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 9d12453c4225455881336b9f31d8e378 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d12453c4225455881336b9f31d8e378 already in progress: creating tablet
I20251026 02:56:46.104339 3997 raft_consensus.cc:493] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.104478 3997 raft_consensus.cc:515] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.105057 3997 leader_election.cc:290] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.105832 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 2337224b44bf4bcc9062d4f8940269ab (DEFAULT_TABLE table=default.Table_a86131ef9a344c92952a42fed41f843e [id=8028f045ad6043608a9cddd9e77ea897]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.106221 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2337224b44bf4bcc9062d4f8940269ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.108685 2100 tablet_service.cc:1505] Processing CreateTablet for tablet d1c7f5868bbe4778970701b4d9fb70e5 (DEFAULT_TABLE table=default.Table_51410f2fee7b4ac595de28ae74467f89 [id=f74f82355d9741fe926b7baf971a0844]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.109076 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1c7f5868bbe4778970701b4d9fb70e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.112269 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 7a17672e74f04a94a534232312da8213 (DEFAULT_TABLE table=default.Table_a9e832f0f3764a788f4a90cb03b5b19b [id=64f3604bb69f45c088fb8e325c036dd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.112638 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a17672e74f04a94a534232312da8213. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:45.933619 3391 rpcz_store.cc:276] 1026 02:56:38.050954 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:38.051031 (+ 77us) service_pool.cc:225] Handling call
1026 02:56:44.687203 (+6636172us) raft_consensus.cc:1471] Updating replica for 530 ops
1026 02:56:44.884895 (+197692us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 02:56:44.884906 (+ 11us) raft_consensus.cc:1539] Triggering prepare for 530 ops
1026 02:56:44.887901 (+ 2995us) write_op.cc:270] Start()
1026 02:56:44.887971 (+ 70us) write_op.cc:276] Timestamp: P: 1761447387288005 usec, L: 0
1026 02:56:44.933784 (+ 45813us) write_op.cc:270] Start()
1026 02:56:44.933829 (+ 45us) write_op.cc:276] Timestamp: P: 1761447387291778 usec, L: 0
1026 02:56:44.934116 (+ 287us) write_op.cc:270] Start()
1026 02:56:44.934167 (+ 51us) write_op.cc:276] Timestamp: P: 1761447387292425 usec, L: 0
1026 02:56:44.934356 (+ 189us) write_op.cc:270] Start()
1026 02:56:44.934379 (+ 23us) write_op.cc:276] Timestamp: P: 1761447387295599 usec, L: 0
1026 02:56:44.934588 (+ 209us) write_op.cc:270] Start()
1026 02:56:44.934611 (+ 23us) write_op.cc:276] Timestamp: P: 1761447387296225 usec, L: 0
1026 02:56:44.934762 (+ 151us) write_op.cc:270] Start()
1026 02:56:44.934783 (+ 21us) write_op.cc:276] Timestamp: P: 1761447387298290 usec, L: 0
1026 02:56:44.934930 (+ 147us) write_op.cc:270] Start()
1026 02:56:44.934986 (+ 56us) write_op.cc:276] Timestamp: P: 1761447387334908 usec, L: 0
1026 02:56:44.935175 (+ 189us) write_op.cc:270] Start()
1026 02:56:44.935199 (+ 24us) write_op.cc:276] Timestamp: P: 1761447387372449 usec, L: 0
1026 02:56:44.935364 (+ 165us) write_op.cc:270] Start()
1026 02:56:44.935402 (+ 38us) write_op.cc:276] Timestamp: P: 1761447387373107 usec, L: 0
1026 02:56:44.935558 (+ 156us) write_op.cc:270] Start()
1026 02:56:44.935579 (+ 21us) write_op.cc:276] Timestamp: P: 1761447387377786 usec, L: 0
1026 02:56:44.935734 (+ 155us) write_op.cc:270] Start()
1026 02:56:44.935765 (+ 31us) write_op.cc:276] Timestamp: P: 1761447387383523 usec, L: 0
1026 02:56:44.935951 (+ 186us) write_op.cc:270] Start()
1026 02:56:44.935973 (+ 22us) write_op.cc:276] Timestamp: P: 1761447387384068 usec, L: 0
1026 02:56:44.936166 (+ 193us) write_op.cc:270] Start()
1026 02:56:44.936193 (+ 27us) write_op.cc:276] Timestamp: P: 1761447387397530 usec, L: 0
1026 02:56:44.936426 (+ 233us) write_op.cc:270] Start()
1026 02:56:44.936452 (+ 26us) write_op.cc:276] Timestamp: P: 1761447387407391 usec, L: 0
1026 02:56:44.936608 (+ 156us) write_op.cc:270] Start()
1026 02:56:44.936631 (+ 23us) write_op.cc:276] Timestamp: P: 1761447387408037 usec, L: 0
1026 02:56:44.936763 (+ 132us) write_op.cc:270] Start()
1026 02:56:44.936782 (+ 19us) write_op.cc:276] Timestamp: P: 1761447387408663 usec, L: 0
1026 02:56:44.936931 (+ 149us) write_op.cc:270] Start()
1026 02:56:44.936963 (+ 32us) write_op.cc:276] Timestamp: P: 1761447387409166 usec, L: 0
1026 02:56:44.937158 (+ 195us) write_op.cc:270] Start()
1026 02:56:44.937185 (+ 27us) write_op.cc:276] Timestamp: P: 1761447387409690 usec, L: 0
1026 02:56:44.937342 (+ 157us) write_op.cc:270] Start()
1026 02:56:44.937363 (+ 21us) write_op.cc:276] Timestamp: P: 1761447387440059 usec, L: 0
1026 02:56:44.937498 (+ 135us) write_op.cc:270] Start()
1026 02:56:44.937517 (+ 19us) write_op.cc:276] Timestamp: P: 1761447387442210 usec, L: 0
1026 02:56:44.937648 (+ 131us) write_op.cc:270] Start()
1026 02:56:44.937669 (+ 21us) write_op.cc:276] Timestamp: P: 1761447387450170 usec, L: 0
1026 02:56:44.937805 (+ 136us) write_op.cc:270] Start()
1026 02:56:44.937825 (+ 20us) write_op.cc:276] Timestamp: P: 1761447387468025 usec, L: 0
1026 02:56:44.938030 (+ 205us) write_op.cc:270] Start()
1026 02:56:44.938055 (+ 25us) write_op.cc:276] Timestamp: P: 1761447387517965 usec, L: 0
1026 02:56:44.976543 (+ 38488us) write_op.cc:270] Start()
1026 02:56:44.976596 (+ 53us) write_op.cc:276] Timestamp: P: 1761447387531804 usec, L: 0
1026 02:56:44.976875 (+ 279us) write_op.cc:270] Start()
1026 02:56:44.976916 (+ 41us) write_op.cc:276] Timestamp: P: 1761447387541378 usec, L: 0
1026 02:56:44.977092 (+ 176us) write_op.cc:270] Start()
1026 02:56:44.977119 (+ 27us) write_op.cc:276] Timestamp: P: 1761447387554214 usec, L: 0
1026 02:56:44.977295 (+ 176us) write_op.cc:270] Start()
1026 02:56:44.977347 (+ 52us) write_op.cc:276] Timestamp: P: 1761447387567008 usec, L: 0
1026 02:56:44.977482 (+ 135us) write_op.cc:270] Start()
1026 02:56:44.977501 (+ 19us) write_op.cc:276] Timestamp: P: 1761447387591064 usec, L: 0
1026 02:56:44.977662 (+ 161us) write_op.cc:270] Start()
1026 02:56:44.977686 (+ 24us) write_op.cc:276] Timestamp: P: 1761447387605796 usec, L: 0
1026 02:56:44.977822 (+ 136us) write_op.cc:270] Start()
1026 02:56:44.977842 (+ 20us) write_op.cc:276] Timestamp: P: 1761447387607219 usec, L: 0
1026 02:56:44.977972 (+ 130us) write_op.cc:270] Start()
1026 02:56:44.977990 (+ 18us) write_op.cc:276] Timestamp: P: 1761447387607801 usec, L: 0
1026 02:56:44.978144 (+ 154us) write_op.cc:270] Start()
1026 02:56:44.978163 (+ 19us) write_op.cc:276] Timestamp: P: 1761447387608394 usec, L: 0
1026 02:56:44.978285 (+ 122us) write_op.cc:270] Start()
1026 02:56:44.978302 (+ 17us) write_op.cc:276] Timestamp: P: 1761447387633435 usec, L: 0
1026 02:56:44.978420 (+ 118us) write_op.cc:270] Start()
1026 02:56:44.978441 (+ 21us) write_op.cc:276] Timestamp: P: 1761447387635181 usec, L: 0
1026 02:56:44.978555 (+ 114us) write_op.cc:270] Start()
1026 02:56:44.978574 (+ 19us) write_op.cc:276] Timestamp: P: 1761447387669993 usec, L: 0
1026 02:56:44.978695 (+ 121us) write_op.cc:270] Start()
1026 02:56:44.978713 (+ 18us) write_op.cc:276] Timestamp: P: 1761447387686246 usec, L: 0
1026 02:56:44.978831 (+ 118us) write_op.cc:270] Start()
1026 02:56:44.978867 (+ 36us) write_op.cc:276] Timestamp: P: 1761447387687744 usec, L: 0
1026 02:56:44.979013 (+ 146us) write_op.cc:270] Start()
1026 02:56:44.979032 (+ 19us) write_op.cc:276] Timestamp: P: 1761447387688528 usec, L: 0
1026 02:56:44.979172 (+ 140us) write_op.cc:270] Start()
1026 02:56:44.979190 (+ 18us) write_op.cc:276] Timestamp: P: 1761447387689113 usec, L: 0
1026 02:56:44.979308 (+ 118us) write_op.cc:270] Start()
1026 02:56:44.979326 (+ 18us) write_op.cc:276] Timestamp: P: 1761447387689616 usec, L: 0
1026 02:56:44.979455 (+ 129us) write_op.cc:270] Start()
1026 02:56:44.979473 (+ 18us) write_op.cc:276] Timestamp: P: 1761447387690262 usec, L: 0
1026 02:56:44.979590 (+ 117us) write_op.cc:270] Start()
1026 02:56:44.979608 (+ 18us) write_op.cc:276] Timestamp: P: 1761447387690801 usec, L: 0
1026 02:56:44.979721 (+ 113us) write_op.cc:270] Start()
1026 02:56:44.979739 (+ 18us) write_op.cc:276] Timestamp: P: 1761447387758668 usec, L: 0
1026 02:56:44.979891 (+ 152us) write_op.cc:270] Start()
1026 02:56:44.979910 (+ 19us) write_op.cc:276] Timestamp: P: 1761447387775956 usec, L: 0
1026 02:56:44.980038 (+ 128us) write_op.cc:270] Start()
1026 02:56:44.980056 (+ 18us) write_op.cc:276] Timestamp: P: 1761447387792001 usec, L: 0
1026 02:56:44.980184 (+ 128us) write_op.cc:270] Start()
1026 02:56:44.980202 (+ 18us) write_op.cc:276] Timestamp: P: 1761447387801600 usec, L: 0
1026 02:56:44.980364 (+ 162us) write_op.cc:270] Start()
1026 02:56:44.980383 (+ 19us) write_op.cc:276] Timestamp: P: 1761447387809231 usec, L: 0
1026 02:56:44.980527 (+ 144us) write_op.cc:270] Start()
1026 02:56:44.980546 (+ 19us) write_op.cc:276] Timestamp: P: 1761447387816165 usec, L: 0
1026 02:56:44.980663 (+ 117us) write_op.cc:270] Start()
1026 02:56:44.980682 (+ 19us) write_op.cc:276] Timestamp: P: 1761447387827509 usec, L: 0
1026 02:56:44.980825 (+ 143us) write_op.cc:270] Start()
1026 02:56:44.980844 (+ 19us) write_op.cc:276] Timestamp: P: 1761447387839992 usec, L: 0
1026 02:56:44.981173 (+ 329us) write_op.cc:270] Start()
1026 02:56:44.981205 (+ 32us) write_op.cc:276] Timestamp: P: 1761447387878859 usec, L: 0
1026 02:56:44.981401 (+ 196us) write_op.cc:270] Start()
1026 02:56:44.981429 (+ 28us) write_op.cc:276] Timestamp: P: 1761447387887953 usec, L: 0
1026 02:56:44.981626 (+ 197us) write_op.cc:270] Start()
1026 02:56:44.981654 (+ 28us) write_op.cc:276] Timestamp: P: 1761447387926924 usec, L: 0
1026 02:56:44.990860 (+ 9206us) write_op.cc:270] Start()
1026 02:56:44.990917 (+ 57us) write_op.cc:276] Timestamp: P: 1761447387941597 usec, L: 0
1026 02:56:44.991181 (+ 264us) write_op.cc:270] Start()
1026 02:56:44.991211 (+ 30us) write_op.cc:276] Timestamp: P: 1761447387967042 usec, L: 0
1026 02:56:44.991385 (+ 174us) write_op.cc:270] Start()
1026 02:56:44.991411 (+ 26us) write_op.cc:276] Timestamp: P: 1761447387982422 usec, L: 0
1026 02:56:44.991588 (+ 177us) write_op.cc:270] Start()
1026 02:56:44.991616 (+ 28us) write_op.cc:276] Timestamp: P: 1761447388003234 usec, L: 0
1026 02:56:44.991797 (+ 181us) write_op.cc:270] Start()
1026 02:56:44.991826 (+ 29us) write_op.cc:276] Timestamp: P: 1761447388035405 usec, L: 0
1026 02:56:44.991988 (+ 162us) write_op.cc:270] Start()
1026 02:56:44.992015 (+ 27us) write_op.cc:276] Timestamp: P: 1761447388050067 usec, L: 0
1026 02:56:44.992292 (+ 277us) write_op.cc:270] Start()
1026 02:56:44.992364 (+ 72us) write_op.cc:276] Timestamp: P: 1761447388072484 usec, L: 0
1026 02:56:44.992630 (+ 266us) write_op.cc:270] Start()
1026 02:56:44.992661 (+ 31us) write_op.cc:276] Timestamp: P: 1761447388093292 usec, L: 0
1026 02:56:44.992886 (+ 225us) write_op.cc:270] Start()
1026 02:56:44.992914 (+ 28us) write_op.cc:276] Timestamp: P: 1761447388105582 usec, L: 0
1026 02:56:44.993109 (+ 195us) write_op.cc:270] Start()
1026 02:56:44.993199 (+ 90us) write_op.cc:276] Timestamp: P: 1761447388116066 usec, L: 0
1026 02:56:44.993393 (+ 194us) write_op.cc:270] Start()
1026 02:56:44.993423 (+ 30us) write_op.cc:276] Timestamp: P: 1761447388134397 usec, L: 0
1026 02:56:44.993627 (+ 204us) write_op.cc:270] Start()
1026 02:56:44.993655 (+ 28us) write_op.cc:276] Timestamp: P: 1761447388156712 usec, L: 0
1026 02:56:44.993852 (+ 197us) write_op.cc:270] Start()
1026 02:56:44.993879 (+ 27us) write_op.cc:276] Timestamp: P: 1761447388168442 usec, L: 0
1026 02:56:44.994054 (+ 175us) write_op.cc:270] Start()
1026 02:56:44.994081 (+ 27us) write_op.cc:276] Timestamp: P: 1761447388170022 usec, L: 0
1026 02:56:45.003097 (+ 9016us) write_op.cc:270] Start()
1026 02:56:45.003145 (+ 48us) write_op.cc:276] Timestamp: P: 1761447388171219 usec, L: 0
1026 02:56:45.003345 (+ 200us) write_op.cc:270] Start()
1026 02:56:45.003381 (+ 36us) write_op.cc:276] Timestamp: P: 1761447388171778 usec, L: 0
1026 02:56:45.003568 (+ 187us) write_op.cc:270] Start()
1026 02:56:45.003590 (+ 22us) write_op.cc:276] Timestamp: P: 1761447388172292 usec, L: 0
1026 02:56:45.003724 (+ 134us) write_op.cc:270] Start()
1026 02:56:45.003748 (+ 24us) write_op.cc:276] Timestamp: P: 1761447388196597 usec, L: 0
1026 02:56:45.003903 (+ 155us) write_op.cc:270] Start()
1026 02:56:45.003925 (+ 22us) write_op.cc:276] Timestamp: P: 1761447388197247 usec, L: 0
1026 02:56:45.004061 (+ 136us) write_op.cc:270] Start()
1026 02:56:45.004083 (+ 22us) write_op.cc:276] Timestamp: P: 1761447388197827 usec, L: 0
1026 02:56:45.004343 (+ 260us) write_op.cc:270] Start()
1026 02:56:45.004369 (+ 26us) write_op.cc:276] Timestamp: P: 1761447388198392 usec, L: 0
1026 02:56:45.010855 (+ 6486us) write_op.cc:270] Start()
1026 02:56:45.010906 (+ 51us) write_op.cc:276] Timestamp: P: 1761447388198946 usec, L: 0
1026 02:56:45.011190 (+ 284us) write_op.cc:270] Start()
1026 02:56:45.011221 (+ 31us) write_op.cc:276] Timestamp: P: 1761447388199515 usec, L: 0
1026 02:56:45.011401 (+ 180us) write_op.cc:270] Start()
1026 02:56:45.011430 (+ 29us) write_op.cc:276] Timestamp: P: 1761447388239405 usec, L: 0
1026 02:56:45.011622 (+ 192us) write_op.cc:270] Start()
1026 02:56:45.011651 (+ 29us) write_op.cc:276] Timestamp: P: 1761447388246873 usec, L: 0
1026 02:56:45.011824 (+ 173us) write_op.cc:270] Start()
1026 02:56:45.011889 (+ 65us) write_op.cc:276] Timestamp: P: 1761447388251300 usec, L: 0
1026 02:56:45.012077 (+ 188us) write_op.cc:270] Start()
1026 02:56:45.012105 (+ 28us) write_op.cc:276] Timestamp: P: 1761447388286502 usec, L: 0
1026 02:56:45.012323 (+ 218us) write_op.cc:270] Start()
1026 02:56:45.012345 (+ 22us) write_op.cc:276] Timestamp: P: 1761447388347781 usec, L: 0
1026 02:56:45.012521 (+ 176us) write_op.cc:270] Start()
1026 02:56:45.012551 (+ 30us) write_op.cc:276] Timestamp: P: 1761447388352905 usec, L: 0
1026 02:56:45.012734 (+ 183us) write_op.cc:270] Start()
1026 02:56:45.012761 (+ 27us) write_op.cc:276] Timestamp: P: 1761447388366496 usec, L: 0
1026 02:56:45.012944 (+ 183us) write_op.cc:270] Start()
1026 02:56:45.012969 (+ 25us) write_op.cc:276] Timestamp: P: 1761447388373865 usec, L: 0
1026 02:56:45.013128 (+ 159us) write_op.cc:270] Start()
1026 02:56:45.013167 (+ 39us) write_op.cc:276] Timestamp: P: 1761447388374733 usec, L: 0
1026 02:56:45.013320 (+ 153us) write_op.cc:270] Start()
1026 02:56:45.013344 (+ 24us) write_op.cc:276] Timestamp: P: 1761447388377523 usec, L: 0
1026 02:56:45.013508 (+ 164us) write_op.cc:270] Start()
1026 02:56:45.013527 (+ 19us) write_op.cc:276] Timestamp: P: 1761447388403025 usec, L: 0
1026 02:56:45.013669 (+ 142us) write_op.cc:270] Start()
1026 02:56:45.013706 (+ 37us) write_op.cc:276] Timestamp: P: 1761447388403751 usec, L: 0
1026 02:56:45.013864 (+ 158us) write_op.cc:270] Start()
1026 02:56:45.013888 (+ 24us) write_op.cc:276] Timestamp: P: 1761447388407742 usec, L: 0
1026 02:56:45.014053 (+ 165us) write_op.cc:270] Start()
1026 02:56:45.014078 (+ 25us) write_op.cc:276] Timestamp: P: 1761447388421708 usec, L: 0
1026 02:56:45.014256 (+ 178us) write_op.cc:270] Start()
1026 02:56:45.014280 (+ 24us) write_op.cc:276] Timestamp: P: 1761447388444771 usec, L: 0
1026 02:56:45.014439 (+ 159us) write_op.cc:270] Start()
1026 02:56:45.014464 (+ 25us) write_op.cc:276] Timestamp: P: 1761447388492305 usec, L: 0
1026 02:56:45.014632 (+ 168us) write_op.cc:270] Start()
1026 02:56:45.014656 (+ 24us) write_op.cc:276] Timestamp: P: 1761447388501691 usec, L: 0
1026 02:56:45.014806 (+ 150us) write_op.cc:270] Start()
1026 02:56:45.014831 (+ 25us) write_op.cc:276] Timestamp: P: 1761447388515958 usec, L: 0
1026 02:56:45.014992 (+ 161us) write_op.cc:270] Start()
1026 02:56:45.015015 (+ 23us) write_op.cc:276] Timestamp: P: 1761447388540828 usec, L: 0
1026 02:56:45.015212 (+ 197us) write_op.cc:270] Start()
1026 02:56:45.015239 (+ 27us) write_op.cc:276] Timestamp: P: 1761447388651053 usec, L: 0
1026 02:56:45.015390 (+ 151us) write_op.cc:270] Start()
1026 02:56:45.015414 (+ 24us) write_op.cc:276] Timestamp: P: 1761447388651799 usec, L: 0
1026 02:56:45.035911 (+ 20497us) write_op.cc:270] Start()
1026 02:56:45.035951 (+ 40us) write_op.cc:276] Timestamp: P: 1761447388652414 usec, L: 0
1026 02:56:45.036172 (+ 221us) write_op.cc:270] Start()
1026 02:56:45.036194 (+ 22us) write_op.cc:276] Timestamp: P: 1761447388715990 usec, L: 0
1026 02:56:45.037015 (+ 821us) write_op.cc:270] Start()
1026 02:56:45.037054 (+ 39us) write_op.cc:276] Timestamp: P: 1761447388717026 usec, L: 0
1026 02:56:45.037258 (+ 204us) write_op.cc:270] Start()
1026 02:56:45.037282 (+ 24us) write_op.cc:276] Timestamp: P: 1761447388717557 usec, L: 0
1026 02:56:45.037425 (+ 143us) write_op.cc:270] Start()
1026 02:56:45.037446 (+ 21us) write_op.cc:276] Timestamp: P: 1761447388718227 usec, L: 0
1026 02:56:45.037569 (+ 123us) write_op.cc:270] Start()
1026 02:56:45.037587 (+ 18us) write_op.cc:276] Timestamp: P: 1761447388718952 usec, L: 0
1026 02:56:45.037703 (+ 116us) write_op.cc:270] Start()
1026 02:56:45.037722 (+ 19us) write_op.cc:276] Timestamp: P: 1761447388719533 usec, L: 0
1026 02:56:45.037832 (+ 110us) write_op.cc:270] Start()
1026 02:56:45.037851 (+ 19us) write_op.cc:276] Timestamp: P: 1761447388795925 usec, L: 0
1026 02:56:45.037980 (+ 129us) write_op.cc:270] Start()
1026 02:56:45.037999 (+ 19us) write_op.cc:276] Timestamp: P: 1761447388798427 usec, L: 0
1026 02:56:45.038180 (+ 181us) write_op.cc:270] Start()
1026 02:56:45.038200 (+ 20us) write_op.cc:276] Timestamp: P: 1761447388799388 usec, L: 0
1026 02:56:45.038335 (+ 135us) write_op.cc:270] Start()
1026 02:56:45.038352 (+ 17us) write_op.cc:276] Timestamp: P: 1761447388800085 usec, L: 0
1026 02:56:45.038474 (+ 122us) write_op.cc:270] Start()
1026 02:56:45.038494 (+ 20us) write_op.cc:276] Timestamp: P: 1761447388871059 usec, L: 0
1026 02:56:45.038634 (+ 140us) write_op.cc:270] Start()
1026 02:56:45.038654 (+ 20us) write_op.cc:276] Timestamp: P: 1761447388871886 usec, L: 0
1026 02:56:45.038787 (+ 133us) write_op.cc:270] Start()
1026 02:56:45.038807 (+ 20us) write_op.cc:276] Timestamp: P: 1761447388872589 usec, L: 0
1026 02:56:45.038946 (+ 139us) write_op.cc:270] Start()
1026 02:56:45.038973 (+ 27us) write_op.cc:276] Timestamp: P: 1761447388873087 usec, L: 0
1026 02:56:45.039142 (+ 169us) write_op.cc:270] Start()
1026 02:56:45.039165 (+ 23us) write_op.cc:276] Timestamp: P: 1761447388886034 usec, L: 0
1026 02:56:45.039338 (+ 173us) write_op.cc:270] Start()
1026 02:56:45.039360 (+ 22us) write_op.cc:276] Timestamp: P: 1761447388891687 usec, L: 0
1026 02:56:45.039506 (+ 146us) write_op.cc:270] Start()
1026 02:56:45.039528 (+ 22us) write_op.cc:276] Timestamp: P: 1761447388938041 usec, L: 0
1026 02:56:45.039732 (+ 204us) write_op.cc:270] Start()
1026 02:56:45.039760 (+ 28us) write_op.cc:276] Timestamp: P: 1761447388938949 usec, L: 0
1026 02:56:45.039931 (+ 171us) write_op.cc:270] Start()
1026 02:56:45.039955 (+ 24us) write_op.cc:276] Timestamp: P: 1761447388939808 usec, L: 0
1026 02:56:45.040097 (+ 142us) write_op.cc:270] Start()
1026 02:56:45.040118 (+ 21us) write_op.cc:276] Timestamp: P: 1761447389087144 usec, L: 0
1026 02:56:45.048955 (+ 8837us) write_op.cc:270] Start()
1026 02:56:45.049003 (+ 48us) write_op.cc:276] Timestamp: P: 1761447389087944 usec, L: 0
1026 02:56:45.049267 (+ 264us) write_op.cc:270] Start()
1026 02:56:45.049297 (+ 30us) write_op.cc:276] Timestamp: P: 1761447389088675 usec, L: 0
1026 02:56:45.049467 (+ 170us) write_op.cc:270] Start()
1026 02:56:45.049492 (+ 25us) write_op.cc:276] Timestamp: P: 1761447389089377 usec, L: 0
1026 02:56:45.049642 (+ 150us) write_op.cc:270] Start()
1026 02:56:45.049663 (+ 21us) write_op.cc:276] Timestamp: P: 1761447389090066 usec, L: 0
1026 02:56:45.049801 (+ 138us) write_op.cc:270] Start()
1026 02:56:45.049821 (+ 20us) write_op.cc:276] Timestamp: P: 1761447389114909 usec, L: 0
1026 02:56:45.050048 (+ 227us) write_op.cc:270] Start()
1026 02:56:45.050077 (+ 29us) write_op.cc:276] Timestamp: P: 1761447389145158 usec, L: 0
1026 02:56:45.050294 (+ 217us) write_op.cc:270] Start()
1026 02:56:45.050321 (+ 27us) write_op.cc:276] Timestamp: P: 1761447389147203 usec, L: 0
1026 02:56:45.050518 (+ 197us) write_op.cc:270] Start()
1026 02:56:45.050552 (+ 34us) write_op.cc:276] Timestamp: P: 1761447389148213 usec, L: 0
1026 02:56:45.050701 (+ 149us) write_op.cc:270] Start()
1026 02:56:45.050725 (+ 24us) write_op.cc:276] Timestamp: P: 1761447389220618 usec, L: 0
1026 02:56:45.050868 (+ 143us) write_op.cc:270] Start()
1026 02:56:45.050890 (+ 22us) write_op.cc:276] Timestamp: P: 1761447389236800 usec, L: 0
1026 02:56:45.051064 (+ 174us) write_op.cc:270] Start()
1026 02:56:45.051091 (+ 27us) write_op.cc:276] Timestamp: P: 1761447389237649 usec, L: 0
1026 02:56:45.051260 (+ 169us) write_op.cc:270] Start()
1026 02:56:45.051299 (+ 39us) write_op.cc:276] Timestamp: P: 1761447389238306 usec, L: 0
1026 02:56:45.051445 (+ 146us) write_op.cc:270] Start()
1026 02:56:45.051499 (+ 54us) write_op.cc:276] Timestamp: P: 1761447389238943 usec, L: 0
1026 02:56:45.051620 (+ 121us) write_op.cc:270] Start()
1026 02:56:45.051639 (+ 19us) write_op.cc:276] Timestamp: P: 1761447389239572 usec, L: 0
1026 02:56:45.051756 (+ 117us) write_op.cc:270] Start()
1026 02:56:45.051776 (+ 20us) write_op.cc:276] Timestamp: P: 1761447389268460 usec, L: 0
1026 02:56:45.051897 (+ 121us) write_op.cc:270] Start()
1026 02:56:45.051915 (+ 18us) write_op.cc:276] Timestamp: P: 1761447389289645 usec, L: 0
1026 02:56:45.052099 (+ 184us) write_op.cc:270] Start()
1026 02:56:45.052121 (+ 22us) write_op.cc:276] Timestamp: P: 1761447389339695 usec, L: 0
1026 02:56:45.058201 (+ 6080us) write_op.cc:270] Start()
1026 02:56:45.058245 (+ 44us) write_op.cc:276] Timestamp: P: 1761447389362701 usec, L: 0
1026 02:56:45.058537 (+ 292us) write_op.cc:270] Start()
1026 02:56:45.058569 (+ 32us) write_op.cc:276] Timestamp: P: 1761447389395166 usec, L: 0
1026 02:56:45.058770 (+ 201us) write_op.cc:270] Start()
1026 02:56:45.058797 (+ 27us) write_op.cc:276] Timestamp: P: 1761447389406256 usec, L: 0
1026 02:56:45.068000 (+ 9203us) write_op.cc:270] Start()
1026 02:56:45.068058 (+ 58us) write_op.cc:276] Timestamp: P: 1761447389426865 usec, L: 0
1026 02:56:45.068351 (+ 293us) write_op.cc:270] Start()
1026 02:56:45.068383 (+ 32us) write_op.cc:276] Timestamp: P: 1761447389483871 usec, L: 0
1026 02:56:45.068624 (+ 241us) write_op.cc:270] Start()
1026 02:56:45.068651 (+ 27us) write_op.cc:276] Timestamp: P: 1761447389485650 usec, L: 0
1026 02:56:45.068823 (+ 172us) write_op.cc:270] Start()
1026 02:56:45.068851 (+ 28us) write_op.cc:276] Timestamp: P: 1761447389486511 usec, L: 0
1026 02:56:45.069013 (+ 162us) write_op.cc:270] Start()
1026 02:56:45.069041 (+ 28us) write_op.cc:276] Timestamp: P: 1761447389535436 usec, L: 0
1026 02:56:45.069212 (+ 171us) write_op.cc:270] Start()
1026 02:56:45.069239 (+ 27us) write_op.cc:276] Timestamp: P: 1761447389546982 usec, L: 0
1026 02:56:45.069392 (+ 153us) write_op.cc:270] Start()
1026 02:56:45.069417 (+ 25us) write_op.cc:276] Timestamp: P: 1761447389647941 usec, L: 0
1026 02:56:45.069570 (+ 153us) write_op.cc:270] Start()
1026 02:56:45.069591 (+ 21us) write_op.cc:276] Timestamp: P: 1761447389661264 usec, L: 0
1026 02:56:45.074539 (+ 4948us) write_op.cc:270] Start()
1026 02:56:45.074592 (+ 53us) write_op.cc:276] Timestamp: P: 1761447389710475 usec, L: 0
1026 02:56:45.074845 (+ 253us) write_op.cc:270] Start()
1026 02:56:45.074875 (+ 30us) write_op.cc:276] Timestamp: P: 1761447389754355 usec, L: 0
1026 02:56:45.075047 (+ 172us) write_op.cc:270] Start()
1026 02:56:45.075076 (+ 29us) write_op.cc:276] Timestamp: P: 1761447389775193 usec, L: 0
1026 02:56:45.075276 (+ 200us) write_op.cc:270] Start()
1026 02:56:45.075294 (+ 18us) write_op.cc:276] Timestamp: P: 1761447389794980 usec, L: 0
1026 02:56:45.075423 (+ 129us) write_op.cc:270] Start()
1026 02:56:45.075442 (+ 19us) write_op.cc:276] Timestamp: P: 1761447389813043 usec, L: 0
1026 02:56:45.075575 (+ 133us) write_op.cc:270] Start()
1026 02:56:45.075593 (+ 18us) write_op.cc:276] Timestamp: P: 1761447389860903 usec, L: 0
1026 02:56:45.075739 (+ 146us) write_op.cc:270] Start()
1026 02:56:45.075756 (+ 17us) write_op.cc:276] Timestamp: P: 1761447389871219 usec, L: 0
1026 02:56:45.075883 (+ 127us) write_op.cc:270] Start()
1026 02:56:45.075902 (+ 19us) write_op.cc:276] Timestamp: P: 1761447389942650 usec, L: 0
1026 02:56:45.076054 (+ 152us) write_op.cc:270] Start()
1026 02:56:45.076074 (+ 20us) write_op.cc:276] Timestamp: P: 1761447390021313 usec, L: 0
1026 02:56:45.076250 (+ 176us) write_op.cc:270] Start()
1026 02:56:45.076278 (+ 28us) write_op.cc:276] Timestamp: P: 1761447390040682 usec, L: 0
1026 02:56:45.076448 (+ 170us) write_op.cc:270] Start()
1026 02:56:45.076469 (+ 21us) write_op.cc:276] Timestamp: P: 1761447390042303 usec, L: 0
1026 02:56:45.076593 (+ 124us) write_op.cc:270] Start()
1026 02:56:45.076612 (+ 19us) write_op.cc:276] Timestamp: P: 1761447390042889 usec, L: 0
1026 02:56:45.076724 (+ 112us) write_op.cc:270] Start()
1026 02:56:45.076742 (+ 18us) write_op.cc:276] Timestamp: P: 1761447390043376 usec, L: 0
1026 02:56:45.076865 (+ 123us) write_op.cc:270] Start()
1026 02:56:45.076884 (+ 19us) write_op.cc:276] Timestamp: P: 1761447390188192 usec, L: 0
1026 02:56:45.077001 (+ 117us) write_op.cc:270] Start()
1026 02:56:45.077023 (+ 22us) write_op.cc:276] Timestamp: P: 1761447390190736 usec, L: 0
1026 02:56:45.077191 (+ 168us) write_op.cc:270] Start()
1026 02:56:45.077213 (+ 22us) write_op.cc:276] Timestamp: P: 1761447390193536 usec, L: 0
1026 02:56:45.077356 (+ 143us) write_op.cc:270] Start()
1026 02:56:45.077377 (+ 21us) write_op.cc:276] Timestamp: P: 1761447390194955 usec, L: 0
1026 02:56:45.077521 (+ 144us) write_op.cc:270] Start()
1026 02:56:45.077546 (+ 25us) write_op.cc:276] Timestamp: P: 1761447390302566 usec, L: 0
1026 02:56:45.077725 (+ 179us) write_op.cc:270] Start()
1026 02:56:45.077747 (+ 22us) write_op.cc:276] Timestamp: P: 1761447390324085 usec, L: 0
1026 02:56:45.077887 (+ 140us) write_op.cc:270] Start()
1026 02:56:45.077906 (+ 19us) write_op.cc:276] Timestamp: P: 1761447390340904 usec, L: 0
1026 02:56:45.078094 (+ 188us) write_op.cc:270] Start()
1026 02:56:45.078181 (+ 87us) write_op.cc:276] Timestamp: P: 1761447390366817 usec, L: 0
1026 02:56:45.078366 (+ 185us) write_op.cc:270] Start()
1026 02:56:45.078387 (+ 21us) write_op.cc:276] Timestamp: P: 1761447390398396 usec, L: 0
1026 02:56:45.078523 (+ 136us) write_op.cc:270] Start()
1026 02:56:45.078542 (+ 19us) write_op.cc:276] Timestamp: P: 1761447390406548 usec, L: 0
1026 02:56:45.078678 (+ 136us) write_op.cc:270] Start()
1026 02:56:45.078696 (+ 18us) write_op.cc:276] Timestamp: P: 1761447390407481 usec, L: 0
1026 02:56:45.078889 (+ 193us) write_op.cc:270] Start()
1026 02:56:45.078917 (+ 28us) write_op.cc:276] Timestamp: P: 1761447390494306 usec, L: 0
1026 02:56:45.079063 (+ 146us) write_op.cc:270] Start()
1026 02:56:45.079084 (+ 21us) write_op.cc:276] Timestamp: P: 1761447390529102 usec, L: 0
1026 02:56:45.079226 (+ 142us) write_op.cc:270] Start()
1026 02:56:45.079246 (+ 20us) write_op.cc:276] Timestamp: P: 1761447390539590 usec, L: 0
1026 02:56:45.079369 (+ 123us) write_op.cc:270] Start()
1026 02:56:45.079388 (+ 19us) write_op.cc:276] Timestamp: P: 1761447390593052 usec, L: 0
1026 02:56:45.079523 (+ 135us) write_op.cc:270] Start()
1026 02:56:45.079542 (+ 19us) write_op.cc:276] Timestamp: P: 1761447390624503 usec, L: 0
1026 02:56:45.079697 (+ 155us) write_op.cc:270] Start()
1026 02:56:45.079725 (+ 28us) write_op.cc:276] Timestamp: P: 1761447390627687 usec, L: 0
1026 02:56:45.079898 (+ 173us) write_op.cc:270] Start()
1026 02:56:45.079920 (+ 22us) write_op.cc:276] Timestamp: P: 1761447390779617 usec, L: 0
1026 02:56:45.080094 (+ 174us) write_op.cc:270] Start()
1026 02:56:45.080116 (+ 22us) write_op.cc:276] Timestamp: P: 1761447390837193 usec, L: 0
1026 02:56:45.090446 (+ 10330us) write_op.cc:270] Start()
1026 02:56:45.090495 (+ 49us) write_op.cc:276] Timestamp: P: 1761447390867382 usec, L: 0
1026 02:56:45.090722 (+ 227us) write_op.cc:270] Start()
1026 02:56:45.090745 (+ 23us) write_op.cc:276] Timestamp: P: 1761447390957424 usec, L: 0
1026 02:56:45.090909 (+ 164us) write_op.cc:270] Start()
1026 02:56:45.090933 (+ 24us) write_op.cc:276] Timestamp: P: 1761447390976576 usec, L: 0
1026 02:56:45.091094 (+ 161us) write_op.cc:270] Start()
1026 02:56:45.091116 (+ 22us) write_op.cc:276] Timestamp: P: 1761447390977253 usec, L: 0
1026 02:56:45.091276 (+ 160us) write_op.cc:270] Start()
1026 02:56:45.091300 (+ 24us) write_op.cc:276] Timestamp: P: 1761447390977773 usec, L: 0
1026 02:56:45.091464 (+ 164us) write_op.cc:270] Start()
1026 02:56:45.091494 (+ 30us) write_op.cc:276] Timestamp: P: 1761447390978360 usec, L: 0
1026 02:56:45.091643 (+ 149us) write_op.cc:270] Start()
1026 02:56:45.091667 (+ 24us) write_op.cc:276] Timestamp: P: 1761447390978861 usec, L: 0
1026 02:56:45.091820 (+ 153us) write_op.cc:270] Start()
1026 02:56:45.091845 (+ 25us) write_op.cc:276] Timestamp: P: 1761447391020718 usec, L: 0
1026 02:56:45.092019 (+ 174us) write_op.cc:270] Start()
1026 02:56:45.092044 (+ 25us) write_op.cc:276] Timestamp: P: 1761447391098992 usec, L: 0
1026 02:56:45.092198 (+ 154us) write_op.cc:270] Start()
1026 02:56:45.092241 (+ 43us) write_op.cc:276] Timestamp: P: 1761447391175552 usec, L: 0
1026 02:56:45.092395 (+ 154us) write_op.cc:270] Start()
1026 02:56:45.092419 (+ 24us) write_op.cc:276] Timestamp: P: 1761447391177096 usec, L: 0
1026 02:56:45.092597 (+ 178us) write_op.cc:270] Start()
1026 02:56:45.092619 (+ 22us) write_op.cc:276] Timestamp: P: 1761447391177667 usec, L: 0
1026 02:56:45.092775 (+ 156us) write_op.cc:270] Start()
1026 02:56:45.092810 (+ 35us) write_op.cc:276] Timestamp: P: 1761447391178229 usec, L: 0
1026 02:56:45.093025 (+ 215us) write_op.cc:270] Start()
1026 02:56:45.093054 (+ 29us) write_op.cc:276] Timestamp: P: 1761447391178731 usec, L: 0
1026 02:56:45.093285 (+ 231us) write_op.cc:270] Start()
1026 02:56:45.093313 (+ 28us) write_op.cc:276] Timestamp: P: 1761447391179272 usec, L: 0
1026 02:56:45.093513 (+ 200us) write_op.cc:270] Start()
1026 02:56:45.093541 (+ 28us) write_op.cc:276] Timestamp: P: 1761447391179822 usec, L: 0
1026 02:56:45.093712 (+ 171us) write_op.cc:270] Start()
1026 02:56:45.093740 (+ 28us) write_op.cc:276] Timestamp: P: 1761447391208546 usec, L: 0
1026 02:56:45.093910 (+ 170us) write_op.cc:270] Start()
1026 02:56:45.093939 (+ 29us) write_op.cc:276] Timestamp: P: 1761447391209386 usec, L: 0
1026 02:56:45.094127 (+ 188us) write_op.cc:270] Start()
1026 02:56:45.094167 (+ 40us) write_op.cc:276] Timestamp: P: 1761447
I20251026 02:56:46.122040 2377 ts_tablet_manager.cc:1916] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.125815 2378 tablet_bootstrap.cc:492] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:46.125962 2378 ts_tablet_manager.cc:1403] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20251026 02:56:46.126641 2378 raft_consensus.cc:359] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.126672 2081 tablet_service.cc:1505] Processing CreateTablet for tablet f59ea356389346359d66b207c5cdb69f (DEFAULT_TABLE table=default.Table_5068032512e84748bf5f2399a08bfa05 [id=f90ac3d3c76b4e03b3ed2bbe11458a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.126802 2378 raft_consensus.cc:385] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.126848 2378 raft_consensus.cc:740] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.061302 2372 ts_tablet_manager.cc:1929] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:46.127048 2372 log.cc:1199] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/e701d4023fee45d18c48997aab3227c2
I20251026 02:56:46.127104 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f59ea356389346359d66b207c5cdb69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.127045 2378 consensus_queue.cc:260] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.127447 2372 ts_tablet_manager.cc:1950] T e701d4023fee45d18c48997aab3227c2 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:46.127555 2378 ts_tablet_manager.cc:1434] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:46.128451 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 866) took 5821 ms. Trace:
I20251026 02:56:46.128541 2372 rpcz_store.cc:276] 1026 02:56:40.307140 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.307967 (+ 827us) service_pool.cc:225] Handling call
1026 02:56:46.014380 (+5706413us) ts_tablet_manager.cc:1151] Deleting tablet e701d4023fee45d18c48997aab3227c2
1026 02:56:46.014391 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.018622 (+ 4231us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.061193 (+ 42571us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.128417 (+ 67224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5706208}
I20251026 02:56:46.128875 2372 tablet_replica.cc:333] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:46.129052 2372 raft_consensus.cc:2243] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.129287 2372 raft_consensus.cc:2272] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.129915 2372 ts_tablet_manager.cc:1916] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.131628 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 9eedb1c430334c44a868e1932591a258 (DEFAULT_TABLE table=default.Table_57ce21096a194256b1f340c52bc836a4 [id=8b1937f297094dad86750ea6b858fd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.132014 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eedb1c430334c44a868e1932591a258. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.135653 2376 ts_tablet_manager.cc:1929] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:46.135758 2376 log.cc:1199] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/dba102838945453cbf44e1cfe505a52c
I20251026 02:56:46.136080 2376 ts_tablet_manager.cc:1950] T dba102838945453cbf44e1cfe505a52c P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:46.136440 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 858) took 4523 ms. Trace:
I20251026 02:56:46.136515 2376 rpcz_store.cc:276] 1026 02:56:41.612707 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.612789 (+ 82us) service_pool.cc:225] Handling call
1026 02:56:46.000927 (+4388138us) ts_tablet_manager.cc:1151] Deleting tablet dba102838945453cbf44e1cfe505a52c
1026 02:56:46.000936 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.041374 (+ 40438us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.135542 (+ 94168us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.136416 (+ 874us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4387938}
I20251026 02:56:46.136897 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 859) took 4492 ms. Trace:
I20251026 02:56:46.136976 2376 rpcz_store.cc:276] 1026 02:56:41.644552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.644639 (+ 87us) service_pool.cc:225] Handling call
1026 02:56:46.136776 (+4492137us) ts_tablet_manager.cc:1151] Deleting tablet 887dc91626664f23b39cf70a6bb9381e
1026 02:56:46.136785 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.136881 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4491900}
I20251026 02:56:46.137308 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 860) took 4456 ms. Trace:
I20251026 02:56:46.137390 2376 rpcz_store.cc:276] 1026 02:56:41.681024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.681088 (+ 64us) service_pool.cc:225] Handling call
1026 02:56:46.137177 (+4456089us) ts_tablet_manager.cc:1151] Deleting tablet ade0f7de597049ffb45274d60c9c0932
1026 02:56:46.137203 (+ 26us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.137291 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4455920}
I20251026 02:56:46.137843 2378 tablet_bootstrap.cc:492] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:46.139428 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 7a17672e74f04a94a534232312da8213 (DEFAULT_TABLE table=default.Table_a9e832f0f3764a788f4a90cb03b5b19b [id=64f3604bb69f45c088fb8e325c036dd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.139478 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 1ad6906bb58a4d5fbbb64c5bc6acef8a (DEFAULT_TABLE table=default.Table_3447a540091c4190a201673b0fc1fa4a [id=21690d84488a400db7536dd15ebb669e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.139675 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 9eedb1c430334c44a868e1932591a258 (DEFAULT_TABLE table=default.Table_57ce21096a194256b1f340c52bc836a4 [id=8b1937f297094dad86750ea6b858fd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.139776 1944 tablet_service.cc:1505] Processing CreateTablet for tablet f59ea356389346359d66b207c5cdb69f (DEFAULT_TABLE table=default.Table_5068032512e84748bf5f2399a08bfa05 [id=f90ac3d3c76b4e03b3ed2bbe11458a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.140076 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 1ad6906bb58a4d5fbbb64c5bc6acef8a (DEFAULT_TABLE table=default.Table_3447a540091c4190a201673b0fc1fa4a [id=21690d84488a400db7536dd15ebb669e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.140570 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad6906bb58a4d5fbbb64c5bc6acef8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.140645 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet dba102838945453cbf44e1cfe505a52c (table default.Table_8ffce6cf6b2d44aa8a9459e834de1d80 [id=92310ee99b5c428c9773b0f7a5462a4d]) successfully deleted
W20251026 02:56:46.140910 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 887dc91626664f23b39cf70a6bb9381e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 887dc91626664f23b39cf70a6bb9381e already in progress: creating tablet
W20251026 02:56:46.141132 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ade0f7de597049ffb45274d60c9c0932 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ade0f7de597049ffb45274d60c9c0932 already in progress: creating tablet
I20251026 02:56:46.141332 3997 raft_consensus.cc:493] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.141448 3997 raft_consensus.cc:515] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.142073 3997 leader_election.cc:290] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.144420 2240 tablet_service.cc:1505] Processing CreateTablet for tablet f59ea356389346359d66b207c5cdb69f (DEFAULT_TABLE table=default.Table_5068032512e84748bf5f2399a08bfa05 [id=f90ac3d3c76b4e03b3ed2bbe11458a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.144536 4003 raft_consensus.cc:493] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.144532 1955 tablet_service.cc:1505] Processing CreateTablet for tablet d1c7f5868bbe4778970701b4d9fb70e5 (DEFAULT_TABLE table=default.Table_51410f2fee7b4ac595de28ae74467f89 [id=f74f82355d9741fe926b7baf971a0844]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.144626 4003 raft_consensus.cc:515] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.144791 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f59ea356389346359d66b207c5cdb69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.145149 4003 leader_election.cc:290] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.145301 4003 consensus_queue.cc:1048] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:46.145808 2224 tablet_service.cc:1505] Processing CreateTablet for tablet d1c7f5868bbe4778970701b4d9fb70e5 (DEFAULT_TABLE table=default.Table_51410f2fee7b4ac595de28ae74467f89 [id=f74f82355d9741fe926b7baf971a0844]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.146113 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1c7f5868bbe4778970701b4d9fb70e5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:46.147943 1629 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 46524.6us
I20251026 02:56:46.148447 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 7a17672e74f04a94a534232312da8213 (DEFAULT_TABLE table=default.Table_a9e832f0f3764a788f4a90cb03b5b19b [id=64f3604bb69f45c088fb8e325c036dd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.148775 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a17672e74f04a94a534232312da8213. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.150866 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 075f6ca8787943a09443863c544a1284
I20251026 02:56:46.151345 1661 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39910 (request call id 1229) took 1140 ms. Trace:
I20251026 02:56:46.151469 1661 rpcz_store.cc:276] 1026 02:56:45.010945 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.475104 (+464159us) service_pool.cc:225] Handling call
1026 02:56:45.475648 (+ 544us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1026 02:56:45.475804 (+ 156us) catalog_manager.cc:2815] Modifying in-memory table state
1026 02:56:45.475868 (+ 64us) catalog_manager.cc:2824] Locking tablets
1026 02:56:46.039523 (+563655us) catalog_manager.cc:2838] Removing table and tablets from system table
1026 02:56:46.150666 (+111143us) catalog_manager.cc:2856] Removing table from by-name map
1026 02:56:46.150751 (+ 85us) catalog_manager.cc:2872] Committing in-memory state
1026 02:56:46.150789 (+ 38us) catalog_manager.cc:2876] Aborting table tasks
1026 02:56:46.151311 (+ 522us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:46.040573 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:46.040670 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:46.040675 (+ 5us) write_op.cc:435] Acquired schema lock
1026 02:56:46.040677 (+ 2us) tablet.cc:660] Decoding operations
1026 02:56:46.040788 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:46.040809 (+ 21us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:46.040814 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1026 02:56:46.040913 (+ 99us) tablet.cc:699] Row locks acquired
1026 02:56:46.040916 (+ 3us) write_op.cc:260] PREPARE: finished
1026 02:56:46.040949 (+ 33us) write_op.cc:270] Start()
1026 02:56:46.040978 (+ 29us) write_op.cc:276] Timestamp: P: 1761447406040922 usec, L: 0
1026 02:56:46.040981 (+ 3us) op_driver.cc:348] REPLICATION: starting
1026 02:56:46.063722 (+ 22741us) log.cc:844] Serialized 725 byte log entry
1026 02:56:46.063913 (+ 191us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x616000041254. stack: 00007f54638f59f6 00007f54638f57b0 00007f54628d13ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f546f790862 00007f5470b732a5 00007f5470b71dc9 00007f5470b7a1f0 00007f5470b79ffd 0000000000a8867c 00007f546393bd7f 00007f546393f04b 00007f546393eec8 0000000000a8867c
1026 02:56:46.149380 (+ 85467us) op_driver.cc:464] REPLICATION: finished
1026 02:56:46.149520 (+ 140us) write_op.cc:301] APPLY: starting
1026 02:56:46.149536 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:46.149583 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:46.149585 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:46.149653 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:46.149686 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1026 02:56:46.149694 (+ 8us) write_op.cc:312] APPLY: finished
1026 02:56:46.149777 (+ 83us) log.cc:844] Serialized 29 byte log entry
1026 02:56:46.149839 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:46.149857 (+ 18us) write_op.cc:454] Released schema lock
1026 02:56:46.149864 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:46.149933 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:46.149937 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":94,"apply.run_cpu_time_us":404,"apply.run_wall_time_us":404,"num_ops":2,"prepare.queue_time_us":444,"prepare.run_cpu_time_us":1220,"prepare.run_wall_time_us":23404,"raft.queue_time_us":1375,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":823,"replication_time_us":108371,"spinlock_wait_cycles":51410688}]]}
I20251026 02:56:46.152503 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 142043b5d558490c9399626c3764ff7e
I20251026 02:56:46.152916 1660 catalog_manager.cc:6719] ReplaceTablet: tablet 142043b5d558490c9399626c3764ff7e deleted and replaced by tablet f2febe856ba04f549c66ea7d95f54482
I20251026 02:56:46.153020 1660 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39910 (request call id 1228) took 1211 ms. Trace:
I20251026 02:56:46.153110 1660 rpcz_store.cc:276] 1026 02:56:44.941426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.414495 (+473069us) service_pool.cc:225] Handling call
1026 02:56:45.423951 (+ 9456us) spinlock_profiling.cc:243] Waited 9 ms on lock 0x616000041254. stack: 00007f54638f59f6 00007f54638f57b0 00007f54628d13ad 0000000000983f04 0000000000983ecd 00007f5472bf36bd 00007f5472bf37b3 00007f546f7be441 00007f54774749dd 00007f547760dfab 00007f546e124107 00007f546e123e35 00007f546a89127e 00007f546a89015e 00007f546a896395 00007f546a896cbf
1026 02:56:46.152992 (+729041us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:46.064005 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:46.064103 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:46.064125 (+ 22us) write_op.cc:435] Acquired schema lock
1026 02:56:46.064127 (+ 2us) tablet.cc:660] Decoding operations
1026 02:56:46.064251 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:46.064261 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:46.064265 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1026 02:56:46.064342 (+ 77us) tablet.cc:699] Row locks acquired
1026 02:56:46.064346 (+ 4us) write_op.cc:260] PREPARE: finished
1026 02:56:46.064380 (+ 34us) write_op.cc:270] Start()
1026 02:56:46.064414 (+ 34us) write_op.cc:276] Timestamp: P: 1761447406064351 usec, L: 0
1026 02:56:46.064418 (+ 4us) op_driver.cc:348] REPLICATION: starting
1026 02:56:46.064599 (+ 181us) log.cc:844] Serialized 622 byte log entry
1026 02:56:46.149453 (+ 84854us) op_driver.cc:464] REPLICATION: finished
1026 02:56:46.150005 (+ 552us) write_op.cc:301] APPLY: starting
1026 02:56:46.150014 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:46.150044 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:46.150070 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:46.150163 (+ 93us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:46.150204 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:46.150212 (+ 8us) write_op.cc:312] APPLY: finished
1026 02:56:46.150261 (+ 49us) log.cc:844] Serialized 29 byte log entry
1026 02:56:46.150298 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:46.150310 (+ 12us) write_op.cc:454] Released schema lock
1026 02:56:46.150315 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":20708224,"child_traces":[["op",{"apply.queue_time_us":523,"apply.run_cpu_time_us":361,"apply.run_wall_time_us":361,"num_ops":2,"prepare.queue_time_us":21809,"prepare.run_cpu_time_us":672,"prepare.run_wall_time_us":672,"raft.queue_time_us":1150,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":85010}]]}
I20251026 02:56:46.154466 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 142043b5d558490c9399626c3764ff7e with delete_type TABLET_DATA_DELETED (replaced by tablet f2febe856ba04f549c66ea7d95f54482) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.155349 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_bc05ce655d584c6480d446730692ceeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.155953 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc05ce655d584c6480d446730692ceeb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.157296 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_234823813b084151a8c49ab91bf0d8f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.157810 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_234823813b084151a8c49ab91bf0d8f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.159211 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 075f6ca8787943a09443863c544a1284 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:45 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:46.159415 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet 142043b5d558490c9399626c3764ff7e with delete_type TABLET_DATA_DELETED (replaced by tablet f2febe856ba04f549c66ea7d95f54482) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:46.159857 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 586bd6e40a7a4785b71b2a3d542e2411
I20251026 02:56:46.159840 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c1e63ac8c5645f3ba1578792c0a0dae" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.160162 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87895fb88c284a4f8d7c58a5ee48ba8e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.160527 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fb2ff87cf0248a0a4a77f2733ef867b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.160593 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_60141a907ea64450acd7914381c35889"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.161070 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60141a907ea64450acd7914381c35889 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:46.161195 2053 leader_election.cc:343] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:46.161439 2053 leader_election.cc:343] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.162557 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 586bd6e40a7a4785b71b2a3d542e2411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.163214 4024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:46.163389 4024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: This master's current role is: LEADER
I20251026 02:56:46.164640 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad6906bb58a4d5fbbb64c5bc6acef8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.164286 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 075f6ca8787943a09443863c544a1284 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:45 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.166595 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 9eedb1c430334c44a868e1932591a258 (DEFAULT_TABLE table=default.Table_57ce21096a194256b1f340c52bc836a4 [id=8b1937f297094dad86750ea6b858fd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.166738 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 142043b5d558490c9399626c3764ff7e with delete_type TABLET_DATA_DELETED (replaced by tablet f2febe856ba04f549c66ea7d95f54482) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.166965 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87895fb88c284a4f8d7c58a5ee48ba8e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:46.167039 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 586bd6e40a7a4785b71b2a3d542e2411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.167219 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 075f6ca8787943a09443863c544a1284 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:45 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.167320 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fb2ff87cf0248a0a4a77f2733ef867b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:46.167466 2262 raft_consensus.cc:2468] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:46.166972 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eedb1c430334c44a868e1932591a258. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:46.167627 2053 leader_election.cc:343] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.167734 2053 leader_election.cc:304] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.168406 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eedb1c430334c44a868e1932591a258. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.169107 3988 raft_consensus.cc:2749] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:46.169476 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c1e63ac8c5645f3ba1578792c0a0dae" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:46.169643 2109 raft_consensus.cc:2468] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:46.170361 2194 leader_election.cc:343] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.171829 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f59ea356389346359d66b207c5cdb69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.175307 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a17672e74f04a94a534232312da8213. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.179057 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1c7f5868bbe4778970701b4d9fb70e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.179924 3997 raft_consensus.cc:493] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.180073 3997 raft_consensus.cc:515] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.180863 3997 leader_election.cc:290] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.181696 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a345fa5edbd48d2891b8b40b09cb3f8" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.182230 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet 586bd6e40a7a4785b71b2a3d542e2411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:46.182404 2053 leader_election.cc:343] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.183182 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a345fa5edbd48d2891b8b40b09cb3f8" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:46.183395 2053 leader_election.cc:304] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.184104 3997 raft_consensus.cc:2804] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:46.184203 3997 raft_consensus.cc:493] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:46.184340 3997 raft_consensus.cc:3060] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.191043 2195 leader_election.cc:304] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.191696 3893 raft_consensus.cc:2804] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:46.191797 3893 raft_consensus.cc:493] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:46.191859 3893 raft_consensus.cc:3060] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.194460 3893 raft_consensus.cc:515] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.194808 2378 tablet_bootstrap.cc:654] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:46.195186 2053 leader_election.cc:343] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.195325 2053 leader_election.cc:304] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.195561 3988 raft_consensus.cc:2749] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:46.199761 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 5bae7b36619e4324a68638736ec9e10a (DEFAULT_TABLE table=default.Table_7e21560192e242298794bb889dfcc622 [id=5c89dedcf7bf4bf0836cacde3d6ccf0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.200129 4009 raft_consensus.cc:493] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:46.200219 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bae7b36619e4324a68638736ec9e10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.200317 4009 raft_consensus.cc:515] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.200982 4009 leader_election.cc:290] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.202780 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 5bae7b36619e4324a68638736ec9e10a (DEFAULT_TABLE table=default.Table_7e21560192e242298794bb889dfcc622 [id=5c89dedcf7bf4bf0836cacde3d6ccf0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.206512 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dbe8002df6c4b64b8ec8260ca6b9cf5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.206745 1975 raft_consensus.cc:2468] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:46.207460 2194 leader_election.cc:304] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:46.208079 4009 raft_consensus.cc:2804] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 02:56:46.208185 4009 raft_consensus.cc:493] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:46.208266 4009 raft_consensus.cc:3060] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:46.210987 4009 raft_consensus.cc:515] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.211613 4009 leader_election.cc:290] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.212047 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bae7b36619e4324a68638736ec9e10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.213032 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 861) took 4492 ms. Trace:
I20251026 02:56:46.213117 2376 rpcz_store.cc:276] 1026 02:56:41.720556 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.720642 (+ 86us) service_pool.cc:225] Handling call
1026 02:56:46.137551 (+4416909us) ts_tablet_manager.cc:1151] Deleting tablet ecbed9dd56e94cad8d7387155b934c50
1026 02:56:46.153908 (+ 16357us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.213014 (+ 59106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4416670}
I20251026 02:56:46.213460 2376 tablet_replica.cc:333] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:46.213647 2376 raft_consensus.cc:2243] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.213804 2376 raft_consensus.cc:2272] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.214167 3997 raft_consensus.cc:515] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.214344 2376 ts_tablet_manager.cc:1916] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.214813 3997 leader_election.cc:290] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.218514 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 1ad6906bb58a4d5fbbb64c5bc6acef8a (DEFAULT_TABLE table=default.Table_3447a540091c4190a201673b0fc1fa4a [id=21690d84488a400db7536dd15ebb669e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.220724 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fb2ff87cf0248a0a4a77f2733ef867b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:46.220919 2262 raft_consensus.cc:3060] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.223652 2262 raft_consensus.cc:2468] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:46.229578 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7bef60e3a10843b4be2022ab4b64be68
I20251026 02:56:46.230746 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet e4c149e8b7f9420c8dba56da2e21f3f9
I20251026 02:56:46.233850 2100 tablet_service.cc:1505] Processing CreateTablet for tablet cac6505fd655412ebc241b546e2a49bc (DEFAULT_TABLE table=default.Table_137bc0aed1ba4f0ebe2575519bea40a8 [id=9b0f9f0b74624edaafdc79a95bd68b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.234311 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac6505fd655412ebc241b546e2a49bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.236577 2098 tablet_service.cc:1505] Processing CreateTablet for tablet c62aaa454551417f904a4198edfd95d1 (DEFAULT_TABLE table=default.Table_94dc93bcf0ca4d58bd2ab854fba805a5 [id=48561e815ee6425983c0187537c95eec]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:46.236815 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ecbed9dd56e94cad8d7387155b934c50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecbed9dd56e94cad8d7387155b934c50 already in progress: creating tablet
I20251026 02:56:46.237006 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62aaa454551417f904a4198edfd95d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.239004 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet e4c149e8b7f9420c8dba56da2e21f3f9 with delete_type TABLET_DATA_DELETED (replaced by tablet a322ec2fa36448e3b66a97d50d4392a1) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.239543 2053 leader_election.cc:304] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:46.239778 3988 raft_consensus.cc:2804] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:46.239864 3988 raft_consensus.cc:697] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:46.240064 3988 consensus_queue.cc:237] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.241345 1662 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (table default.Table_30611e161c274e8b99bd1251e5e527aa [id=b5d91d3e8a2643c09fee85976972391a]). Expected version 1 got 0
I20251026 02:56:46.245849 1662 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 386e1bd4635b4a7e8e7f249b577fd978 (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]). Expected version 2 got 0
I20251026 02:56:46.246132 1656 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_52417f25dbdb4676a5aaaca9bf46962c [id=cc7a8c37a3b342d988872e3baa16a26d] alter complete (version 1)
I20251026 02:56:46.247843 2377 ts_tablet_manager.cc:1929] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:46.247967 2377 log.cc:1199] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7f22c6c163c34ed48a0b7eeb5a574e76
I20251026 02:56:46.248306 2377 ts_tablet_manager.cc:1950] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:46.248893 2375 tablet_bootstrap.cc:492] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:46.251003 2375 tablet_bootstrap.cc:654] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:46.243551 1659 catalog_manager.cc:6719] ReplaceTablet: tablet e4c149e8b7f9420c8dba56da2e21f3f9 deleted and replaced by tablet a322ec2fa36448e3b66a97d50d4392a1
I20251026 02:56:46.253261 2378 tablet_bootstrap.cc:492] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:46.253350 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bef60e3a10843b4be2022ab4b64be68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:45 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.253407 2378 ts_tablet_manager.cc:1403] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.116s user 0.002s sys 0.001s
I20251026 02:56:46.253696 3893 leader_election.cc:290] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.254326 2378 raft_consensus.cc:359] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.254498 2378 raft_consensus.cc:385] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.254544 2378 raft_consensus.cc:740] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.254575 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fb2ff87cf0248a0a4a77f2733ef867b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:46.254729 2378 consensus_queue.cc:260] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.255190 2378 ts_tablet_manager.cc:1434] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:46.257251 1956 tablet_service.cc:1505] Processing CreateTablet for tablet d48c9c2f2b604f39bc6f21bc054c0fad (DEFAULT_TABLE table=default.Table_21830ac7f7ef4cfdb68626b890c460b6 [id=2fcb2c98ea654a0da7ae2f5e8ab6b36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.257531 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 66bd386b3e4f4b48a897feb884330933 (DEFAULT_TABLE table=default.Table_fa73267dd5574a2cbc440226ab6cbb40 [id=9d16a95c505a4048a6114e5d2d1bf965]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.257802 1951 tablet_service.cc:1505] Processing CreateTablet for tablet c62aaa454551417f904a4198edfd95d1 (DEFAULT_TABLE table=default.Table_94dc93bcf0ca4d58bd2ab854fba805a5 [id=48561e815ee6425983c0187537c95eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.257879 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet e4c149e8b7f9420c8dba56da2e21f3f9 with delete_type TABLET_DATA_DELETED (replaced by tablet a322ec2fa36448e3b66a97d50d4392a1) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:46.258078 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bef60e3a10843b4be2022ab4b64be68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:45 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:46.258098 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 36e70ed89e8c422d8448ad3e0dd8b832 (DEFAULT_TABLE table=default.Table_860f88c3c76242d28c07e73bcbb7d077 [id=fd523b355bbd4a78b9e329ab68a6315a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.258966 2233 tablet_service.cc:1505] Processing CreateTablet for tablet d48c9c2f2b604f39bc6f21bc054c0fad (DEFAULT_TABLE table=default.Table_21830ac7f7ef4cfdb68626b890c460b6 [id=2fcb2c98ea654a0da7ae2f5e8ab6b36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.259291 2234 tablet_service.cc:1505] Processing CreateTablet for tablet cac6505fd655412ebc241b546e2a49bc (DEFAULT_TABLE table=default.Table_137bc0aed1ba4f0ebe2575519bea40a8 [id=9b0f9f0b74624edaafdc79a95bd68b1f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:46.259411 2195 leader_election.cc:343] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 3dbe8002df6c4b64b8ec8260ca6b9cf5
I20251026 02:56:46.259446 2232 tablet_service.cc:1505] Processing CreateTablet for tablet c62aaa454551417f904a4198edfd95d1 (DEFAULT_TABLE table=default.Table_94dc93bcf0ca4d58bd2ab854fba805a5 [id=48561e815ee6425983c0187537c95eec]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:46.259578 2195 leader_election.cc:343] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 3dbe8002df6c4b64b8ec8260ca6b9cf5
I20251026 02:56:46.259769 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 66bd386b3e4f4b48a897feb884330933 (DEFAULT_TABLE table=default.Table_fa73267dd5574a2cbc440226ab6cbb40 [id=9d16a95c505a4048a6114e5d2d1bf965]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.259968 2231 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bef60e3a10843b4be2022ab4b64be68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:45 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.259310 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet e4c149e8b7f9420c8dba56da2e21f3f9 with delete_type TABLET_DATA_DELETED (replaced by tablet a322ec2fa36448e3b66a97d50d4392a1) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.258997 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 36e70ed89e8c422d8448ad3e0dd8b832 (DEFAULT_TABLE table=default.Table_860f88c3c76242d28c07e73bcbb7d077 [id=fd523b355bbd4a78b9e329ab68a6315a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.254621 2372 ts_tablet_manager.cc:1929] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:46.260418 2372 log.cc:1199] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/9b9a5daa36b547a7888d38dc3329eaf2
I20251026 02:56:46.260560 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c1e63ac8c5645f3ba1578792c0a0dae" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:46.260722 2372 ts_tablet_manager.cc:1950] T 9b9a5daa36b547a7888d38dc3329eaf2 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:46.260738 2109 raft_consensus.cc:3060] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.261544 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 871) took 5871 ms. Trace:
I20251026 02:56:46.263509 2109 raft_consensus.cc:2468] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:46.258070 2053 leader_election.cc:343] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.267527 2376 ts_tablet_manager.cc:1929] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:46.267676 2376 log.cc:1199] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3c3050e72d42417ab58a05f7e3aa56b3
I20251026 02:56:46.268031 2376 ts_tablet_manager.cc:1950] T 3c3050e72d42417ab58a05f7e3aa56b3 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:46.269737 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 5bae7b36619e4324a68638736ec9e10a (DEFAULT_TABLE table=default.Table_7e21560192e242298794bb889dfcc622 [id=5c89dedcf7bf4bf0836cacde3d6ccf0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.274773 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4f8a62d4db7e4215938585f18149e939" } new_table_name: "default.Table_04a58bdc201c45eaafcdbcfc8ccf93bd" modify_external_catalogs: true
I20251026 02:56:46.273885 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e6e84b24da5b4c8eb22ab71b440d5a0f" } new_table_name: "default.Table_62eeaf16959a496aa0f0bb2f026a7970" modify_external_catalogs: true
I20251026 02:56:46.276237 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dbe8002df6c4b64b8ec8260ca6b9cf5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:46.276440 1976 raft_consensus.cc:3060] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 2
I20251026 02:56:46.277431 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e372c60e85634713bc49acb9d2fbee32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:46.277737 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c1e63ac8c5645f3ba1578792c0a0dae" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:46.278056 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e372c60e85634713bc49acb9d2fbee32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.279518 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_73427fdee7c040a0a8dc618e777f1182"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.280040 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73427fdee7c040a0a8dc618e777f1182 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.281450 1976 raft_consensus.cc:2468] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 2.
I20251026 02:56:46.281555 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6798c68a24044bce9d44d8f21417ebf0" } new_table_name: "default.Table_35a3e6169a8346a497f84b70512a1fcb" modify_external_catalogs: true
I20251026 02:56:46.283131 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7a6f36e22379480f957ea030136b008f" } new_table_name: "default.Table_2d5b5249ce474d81ae1401d1aed8013a" modify_external_catalogs: true
I20251026 02:56:46.284102 2082 tablet_service.cc:1505] Processing CreateTablet for tablet d48c9c2f2b604f39bc6f21bc054c0fad (DEFAULT_TABLE table=default.Table_21830ac7f7ef4cfdb68626b890c460b6 [id=2fcb2c98ea654a0da7ae2f5e8ab6b36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.285993 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e9df6f46f0f3408d9aa07452b0d7e77d" } modify_external_catalogs: true
I20251026 02:56:46.286190 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad6906bb58a4d5fbbb64c5bc6acef8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.286204 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e9df6f46f0f3408d9aa07452b0d7e77d" } modify_external_catalogs: true
I20251026 02:56:46.287943 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 36e70ed89e8c422d8448ad3e0dd8b832 (DEFAULT_TABLE table=default.Table_860f88c3c76242d28c07e73bcbb7d077 [id=fd523b355bbd4a78b9e329ab68a6315a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.288379 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62aaa454551417f904a4198edfd95d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.292384 1655 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_2cf16843709c4e2a9c3632c25c6bbe25 [id=2e17bae10e0b48a492f5ee0bed7d00d0] alter complete (version 2)
I20251026 02:56:46.292896 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 885) took 5236 ms. Trace:
I20251026 02:56:46.293009 2377 rpcz_store.cc:276] 1026 02:56:41.056757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.058174 (+ 1417us) service_pool.cc:225] Handling call
1026 02:56:46.094729 (+5036555us) ts_tablet_manager.cc:1151] Deleting tablet 7f22c6c163c34ed48a0b7eeb5a574e76
1026 02:56:46.094737 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.133787 (+ 39050us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.247718 (+113931us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.292870 (+ 45152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5036273}
I20251026 02:56:46.293396 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 886) took 5234 ms. Trace:
I20251026 02:56:46.293491 2377 rpcz_store.cc:276] 1026 02:56:41.058619 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.058660 (+ 41us) service_pool.cc:225] Handling call
1026 02:56:46.293272 (+5234612us) ts_tablet_manager.cc:1151] Deleting tablet a32b1a6b856c47b993a308db94dac357
1026 02:56:46.293283 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.293380 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5234392}
I20251026 02:56:46.293730 2377 tablet_replica.cc:333] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:46.293931 2377 raft_consensus.cc:2243] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:46.294260 2377 raft_consensus.cc:2272] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.294828 2377 ts_tablet_manager.cc:1916] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.296917 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 209fa71af13142a58efb9f3dab3eecd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.298010 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 862) took 4017 ms. Trace:
W20251026 02:56:46.298211 2194 leader_election.cc:343] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.298446 2194 leader_election.cc:304] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:46.298997 3890 raft_consensus.cc:2804] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 02:56:46.299117 3890 raft_consensus.cc:697] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 2 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:46.298131 2376 rpcz_store.cc:276] 1026 02:56:42.280932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.280996 (+ 64us) service_pool.cc:225] Handling call
1026 02:56:46.213412 (+3932416us) ts_tablet_manager.cc:1151] Deleting tablet 3c3050e72d42417ab58a05f7e3aa56b3
1026 02:56:46.213421 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.245416 (+ 31995us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.267387 (+ 21971us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.297984 (+ 30597us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3932217}
I20251026 02:56:46.299388 2376 tablet_replica.cc:333] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:46.299363 3890 consensus_queue.cc:237] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.299587 2376 raft_consensus.cc:2243] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.299722 2376 raft_consensus.cc:2272] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.299693 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_aeb8c52cba3e49768ced34f8a8b72298" } modify_external_catalogs: true
I20251026 02:56:46.299883 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_aeb8c52cba3e49768ced34f8a8b72298" } modify_external_catalogs: true
I20251026 02:56:46.300349 2376 ts_tablet_manager.cc:1916] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.302544 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d2755d308fc34447b8e72260cb3c0f76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.303130 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2755d308fc34447b8e72260cb3c0f76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.304805 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48c9c2f2b604f39bc6f21bc054c0fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.261646 2372 rpcz_store.cc:276] 1026 02:56:40.389924 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.389968 (+ 44us) service_pool.cc:225] Handling call
1026 02:56:46.128821 (+5738853us) ts_tablet_manager.cc:1151] Deleting tablet 9b9a5daa36b547a7888d38dc3329eaf2
1026 02:56:46.128833 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.243715 (+114882us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.254516 (+ 10801us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.261520 (+ 7004us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5738669}
I20251026 02:56:46.306483 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 867) took 5931 ms. Trace:
I20251026 02:56:46.306612 2372 rpcz_store.cc:276] 1026 02:56:40.375025 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.375081 (+ 56us) service_pool.cc:225] Handling call
1026 02:56:46.306206 (+5931125us) ts_tablet_manager.cc:1151] Deleting tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61
1026 02:56:46.306219 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.306460 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5914129}
I20251026 02:56:46.307154 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 869) took 5931 ms. Trace:
I20251026 02:56:46.307272 2372 rpcz_store.cc:276] 1026 02:56:40.376155 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.376200 (+ 45us) service_pool.cc:225] Handling call
1026 02:56:46.306991 (+5930791us) ts_tablet_manager.cc:1151] Deleting tablet 4e645650709f47eea4243ab64ec31312
1026 02:56:46.307003 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.307138 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5914854}
I20251026 02:56:46.307727 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 868) took 5932 ms. Trace:
I20251026 02:56:46.307870 2372 rpcz_store.cc:276] 1026 02:56:40.375348 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.375494 (+ 146us) service_pool.cc:225] Handling call
1026 02:56:46.307592 (+5932098us) ts_tablet_manager.cc:1151] Deleting tablet 45e79be9b98945c49fb712019b21b91b
1026 02:56:46.307600 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.307692 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5915312}
I20251026 02:56:46.308553 2372 tablet_replica.cc:333] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:46.308818 2372 raft_consensus.cc:2243] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.309060 2372 raft_consensus.cc:2272] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.309998 2372 ts_tablet_manager.cc:1916] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.305495 2378 tablet_bootstrap.cc:492] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:46.305783 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e70ed89e8c422d8448ad3e0dd8b832. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:46.312390 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd9eef082c084267b10163fee196ba6f consensus state has no leader
I20251026 02:56:46.312740 1656 master_service.cc:954] ReplaceTablet: received request to replace tablet ecbed9dd56e94cad8d7387155b934c50 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:46.313369 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66bd386b3e4f4b48a897feb884330933. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.313314 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9cbff204e967439e869756f29385106a" } new_table_name: "default.Table_3827ff5094f14029b7c7f70470594464" modify_external_catalogs: true
I20251026 02:56:46.315660 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 9b9a5daa36b547a7888d38dc3329eaf2 (table default.Table_217ad44ad25140129bf8e39c88b3b593 [id=26a5eae9586f4d00b5ebaf9eb3462272]) successfully deleted
W20251026 02:56:46.315887 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d3ac8d2602d4c2bbb66e3e4cf843d61 already in progress: creating tablet
W20251026 02:56:46.316061 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 4e645650709f47eea4243ab64ec31312 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e645650709f47eea4243ab64ec31312 already in progress: creating tablet
W20251026 02:56:46.316254 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 45e79be9b98945c49fb712019b21b91b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45e79be9b98945c49fb712019b21b91b already in progress: creating tablet
I20251026 02:56:46.317943 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2f7abb9f1aaf4a52b3c0066e12817faf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.318540 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f7abb9f1aaf4a52b3c0066e12817faf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.321089 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 0c6963a62d3949e68662a35d9d528f1b (DEFAULT_TABLE table=default.Table_03342f2363b44f6599f1d4cc6fef2be8 [id=f940fc82956f473580c87805ceac50b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.321177 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 66bd386b3e4f4b48a897feb884330933 (DEFAULT_TABLE table=default.Table_fa73267dd5574a2cbc440226ab6cbb40 [id=9d16a95c505a4048a6114e5d2d1bf965]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.325441 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e70ed89e8c422d8448ad3e0dd8b832. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.325711 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c6963a62d3949e68662a35d9d528f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.327601 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66bd386b3e4f4b48a897feb884330933. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.328516 2376 ts_tablet_manager.cc:1929] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:46.328689 2376 log.cc:1199] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/ae95654d129441e08fcb3f3b355ffc22
I20251026 02:56:46.329051 2376 ts_tablet_manager.cc:1950] T ae95654d129441e08fcb3f3b355ffc22 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:46.329308 2381 tablet_bootstrap.cc:492] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:46.329661 2381 ts_tablet_manager.cc:1403] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.276s user 0.005s sys 0.000s
I20251026 02:56:46.329994 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 863) took 4048 ms. Trace:
I20251026 02:56:46.330122 2376 rpcz_store.cc:276] 1026 02:56:42.281512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.281561 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:46.299345 (+4017784us) ts_tablet_manager.cc:1151] Deleting tablet ae95654d129441e08fcb3f3b355ffc22
1026 02:56:46.299353 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.303389 (+ 4036us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.328412 (+ 25023us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.329972 (+ 1560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4017628}
I20251026 02:56:46.330377 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48c9c2f2b604f39bc6f21bc054c0fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.330619 2376 tablet_replica.cc:333] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:46.330845 2376 raft_consensus.cc:2243] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.331040 2376 raft_consensus.cc:2272] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.331629 2376 ts_tablet_manager.cc:1916] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.331784 2381 raft_consensus.cc:359] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.332361 2381 raft_consensus.cc:385] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.332571 2381 raft_consensus.cc:740] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.333127 2381 consensus_queue.cc:260] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.334676 2381 ts_tablet_manager.cc:1434] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251026 02:56:46.335422 2381 tablet_bootstrap.cc:492] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:46.337626 2195 leader_election.cc:304] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:46.338131 2195 consensus_peers.cc:597] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21b936fa4f6a4ff6af321971a30cccbe. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:46.338433 2376 ts_tablet_manager.cc:1929] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:46.338613 2376 log.cc:1199] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3b7d85a2941545ceaa16b22e8328b4ff
I20251026 02:56:46.339037 2376 ts_tablet_manager.cc:1950] T 3b7d85a2941545ceaa16b22e8328b4ff P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:46.339123 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62aaa454551417f904a4198edfd95d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.339318 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48c9c2f2b604f39bc6f21bc054c0fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.339959 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 864) took 4058 ms. Trace:
I20251026 02:56:46.340091 2376 rpcz_store.cc:276] 1026 02:56:42.281992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.282040 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:46.330570 (+4048530us) ts_tablet_manager.cc:1151] Deleting tablet 3b7d85a2941545ceaa16b22e8328b4ff
1026 02:56:46.330581 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.335169 (+ 4588us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.338344 (+ 3175us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.339938 (+ 1594us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4048388}
I20251026 02:56:46.340605 2376 tablet_replica.cc:333] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:46.340826 2376 raft_consensus.cc:2243] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.341020 2376 raft_consensus.cc:2272] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.341624 2376 ts_tablet_manager.cc:1916] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.342150 1976 raft_consensus.cc:3060] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.344358 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac6505fd655412ebc241b546e2a49bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.344870 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet ec13c5386abe414ba25589e03d5b850a with delete_type TABLET_DATA_DELETED (replaced by tablet 6699b50d87964f6a89c365171253ad25) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:46.345060 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e38f282e8414b7db2a6d5f4a8933e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e0b929f603b4c25b5b971ee9db683e8) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:46.345465 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 0c6963a62d3949e68662a35d9d528f1b (DEFAULT_TABLE table=default.Table_03342f2363b44f6599f1d4cc6fef2be8 [id=f940fc82956f473580c87805ceac50b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.345819 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c6963a62d3949e68662a35d9d528f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.351192 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e70ed89e8c422d8448ad3e0dd8b832. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.352254 1960 tablet_service.cc:1505] Processing CreateTablet for tablet cac6505fd655412ebc241b546e2a49bc (DEFAULT_TABLE table=default.Table_137bc0aed1ba4f0ebe2575519bea40a8 [id=9b0f9f0b74624edaafdc79a95bd68b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.352629 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac6505fd655412ebc241b546e2a49bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.353991 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_191c164a60554219a63000aad6c771b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.354633 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_191c164a60554219a63000aad6c771b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.357851 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66bd386b3e4f4b48a897feb884330933. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.358067 2378 tablet_bootstrap.cc:654] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:46.363189 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3c3050e72d42417ab58a05f7e3aa56b3 (table default.Table_1fba8934d98d4fdb8b2c9a0b5d01e810 [id=b9b139445a9040398621353ad052f3c2]) successfully deleted
I20251026 02:56:46.363745 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet ae95654d129441e08fcb3f3b355ffc22 (table default.Table_8602d75889ea4576ae575b55fe95e940 [id=b0104213af32435589ae19109b7aee7f]) successfully deleted
I20251026 02:56:46.363907 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3b7d85a2941545ceaa16b22e8328b4ff (table default.Table_5ae1979aa25849fe8b780ecc617bb7e8 [id=9f6ac6400247491485d6bef30bf6cf4b]) successfully deleted
I20251026 02:56:46.363973 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bae7b36619e4324a68638736ec9e10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.364694 3888 raft_consensus.cc:2804] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 02:56:46.377593 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88b0cca095514614817346d6eb9014f5 consensus state has no leader
I20251026 02:56:46.380285 4067 raft_consensus.cc:493] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.384845 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_03ccdc56e6a14ea9af51795ee3c3fc08" } modify_external_catalogs: true
I20251026 02:56:46.385021 2376 ts_tablet_manager.cc:1929] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:46.385056 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_03ccdc56e6a14ea9af51795ee3c3fc08" } modify_external_catalogs: true
I20251026 02:56:46.385140 2376 log.cc:1199] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/d65b0d79e2e54af9810022b683c3288b
I20251026 02:56:46.385499 2376 ts_tablet_manager.cc:1950] T d65b0d79e2e54af9810022b683c3288b P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:46.391876 2372 ts_tablet_manager.cc:1929] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:46.392019 2372 log.cc:1199] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/87895fb88c284a4f8d7c58a5ee48ba8e
I20251026 02:56:46.392213 4067 raft_consensus.cc:515] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.392452 2372 ts_tablet_manager.cc:1950] T 87895fb88c284a4f8d7c58a5ee48ba8e P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:46.393119 4067 leader_election.cc:290] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.393178 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 872) took 5993 ms. Trace:
I20251026 02:56:46.393265 2372 rpcz_store.cc:276] 1026 02:56:40.399358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.399423 (+ 65us) service_pool.cc:225] Handling call
1026 02:56:46.308486 (+5909063us) ts_tablet_manager.cc:1151] Deleting tablet 87895fb88c284a4f8d7c58a5ee48ba8e
1026 02:56:46.308494 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.381389 (+ 72895us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.391756 (+ 10367us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.393155 (+ 1399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5908815}
I20251026 02:56:46.393774 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 87895fb88c284a4f8d7c58a5ee48ba8e (table default.Table_42d9349693d44bc5bff21ffa159fd32f [id=f28b682102ae461e96172ab3c0529193]) successfully deleted
I20251026 02:56:46.394177 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 873) took 5984 ms. Trace:
I20251026 02:56:46.394261 2372 rpcz_store.cc:276] 1026 02:56:40.409477 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.409527 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:46.394035 (+5984508us) ts_tablet_manager.cc:1151] Deleting tablet 7096b78402074049b9c8ed1b9fe21f0e
1026 02:56:46.394045 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.394159 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5984149}
W20251026 02:56:46.394701 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7096b78402074049b9c8ed1b9fe21f0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7096b78402074049b9c8ed1b9fe21f0e already in progress: creating tablet
I20251026 02:56:46.395174 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 870) took 6005 ms. Trace:
I20251026 02:56:46.395249 2372 rpcz_store.cc:276] 1026 02:56:40.389781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.409924 (+ 20143us) service_pool.cc:225] Handling call
1026 02:56:46.395072 (+5985148us) ts_tablet_manager.cc:1151] Deleting tablet 8a076043f05c42d9afd9699fb4bec5ba
1026 02:56:46.395082 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.395158 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5984935}
W20251026 02:56:46.395663 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 8a076043f05c42d9afd9699fb4bec5ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a076043f05c42d9afd9699fb4bec5ba already in progress: creating tablet
I20251026 02:56:46.395961 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 876) took 5956 ms. Trace:
I20251026 02:56:46.396045 2372 rpcz_store.cc:276] 1026 02:56:40.439525 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.439591 (+ 66us) service_pool.cc:225] Handling call
1026 02:56:46.395886 (+5956295us) ts_tablet_manager.cc:1151] Deleting tablet ce922040298d4952af13652eeda7a234
1026 02:56:46.395894 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.395945 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5956017}
W20251026 02:56:46.396447 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ce922040298d4952af13652eeda7a234 because the tablet was not found. No further retry: Not found: Tablet not found: ce922040298d4952af13652eeda7a234
I20251026 02:56:46.396698 2372 tablet_replica.cc:333] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:46.396868 2372 raft_consensus.cc:2243] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.397010 2372 raft_consensus.cc:2272] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.397603 2372 ts_tablet_manager.cc:1916] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.400895 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 1807c0737d744d4cab29393e9122ff6b (DEFAULT_TABLE table=default.Table_3fb4e68115eb4b9f98a57da13662a443 [id=ec989ffdd24046fb905f96c691a823d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.401391 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1807c0737d744d4cab29393e9122ff6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.407065 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7eb7fd8131cc4bce81b6cc09c5062fc0" } modify_external_catalogs: true
I20251026 02:56:46.383260 3888 raft_consensus.cc:697] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
W20251026 02:56:46.407292 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7a5020acff04c1d8b5749cdfd7dddfd consensus state has no leader
I20251026 02:56:46.410578 3888 consensus_queue.cc:237] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:46.391827 2194 consensus_peers.cc:597] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dba102838945453cbf44e1cfe505a52c. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:46.411991 2372 ts_tablet_manager.cc:1929] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:46.412135 2372 log.cc:1199] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/c31d2704d1f14ddead69c794cc631ec1
I20251026 02:56:46.412526 2372 ts_tablet_manager.cc:1950] T c31d2704d1f14ddead69c794cc631ec1 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:46.412688 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet c31d2704d1f14ddead69c794cc631ec1 with delete_type TABLET_DATA_DELETED (replaced by tablet de9124fe1c8945d9b9565cb732bef20e) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:46.393290 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 54922.7us
W20251026 02:56:46.413214 2195 consensus_peers.cc:597] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfa68f2c942c43e2a02e9d333bdfa2a2. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:46.413270 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 875) took 5973 ms. Trace:
I20251026 02:56:46.413362 2372 rpcz_store.cc:276] 1026 02:56:40.439396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.439687 (+ 291us) service_pool.cc:225] Handling call
1026 02:56:46.396659 (+5956972us) ts_tablet_manager.cc:1151] Deleting tablet c31d2704d1f14ddead69c794cc631ec1
1026 02:56:46.396666 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.408542 (+ 11876us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.411892 (+ 3350us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.413249 (+ 1357us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48,"tablet-delete.queue_time_us":5956790}
I20251026 02:56:46.413610 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7f22c6c163c34ed48a0b7eeb5a574e76 (table default.Table_0d7b9434a0494a618c25ceebd7f65de8 [id=bb7b13116ad94fa9bd15a1cdb15d207b]) successfully deleted
I20251026 02:56:46.413769 2372 tablet_replica.cc:333] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:46.413954 2372 raft_consensus.cc:2243] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.410432 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7eb7fd8131cc4bce81b6cc09c5062fc0" } modify_external_catalogs: true
I20251026 02:56:46.414106 2372 raft_consensus.cc:2272] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.414115 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet c31d2704d1f14ddead69c794cc631ec1 (table default.Table_b6138cb4ba3c4b70bcec42f1ce989668 [id=9b9d43027de84287b1e2fd35f5e4d5f5]) successfully deleted
I20251026 02:56:46.414495 3997 raft_consensus.cc:493] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.414677 2372 ts_tablet_manager.cc:1916] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.414652 3997 raft_consensus.cc:515] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.415319 3997 leader_election.cc:290] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:46.415787 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet a32b1a6b856c47b993a308db94dac357 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a32b1a6b856c47b993a308db94dac357 already in progress: creating tablet
I20251026 02:56:46.417496 1654 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:39860 (request call id 6) took 1586 ms. Trace:
I20251026 02:56:46.417654 1654 rpcz_store.cc:276] 1026 02:56:44.831174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.417230 (+1586056us) service_pool.cc:225] Handling call
1026 02:56:46.417454 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:46.418378 1654 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:39876 (request call id 12) took 1052 ms. Trace:
I20251026 02:56:46.418699 1654 rpcz_store.cc:276] 1026 02:56:45.366162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.418092 (+1051930us) service_pool.cc:225] Handling call
1026 02:56:46.418330 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:46.421991 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5228f482c0b4fcbbdf8ebc31bb62819" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.422160 2372 ts_tablet_manager.cc:1929] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:46.422297 2372 log.cc:1199] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/ec13c5386abe414ba25589e03d5b850a
I20251026 02:56:46.424201 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5228f482c0b4fcbbdf8ebc31bb62819" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:46.424549 2116 raft_consensus.cc:2468] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:46.425139 2378 tablet_bootstrap.cc:492] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:46.425279 2378 ts_tablet_manager.cc:1403] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.120s user 0.006s sys 0.001s
I20251026 02:56:46.426153 2378 raft_consensus.cc:359] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.426354 2378 raft_consensus.cc:385] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.426412 2378 raft_consensus.cc:740] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.426645 2378 consensus_queue.cc:260] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.427243 2378 ts_tablet_manager.cc:1434] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:46.429080 2378 tablet_bootstrap.cc:492] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:46.430732 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 3e424a9c89e64b1c98ada17fa5c31e9e from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:46.431205 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 1807c0737d744d4cab29393e9122ff6b (DEFAULT_TABLE table=default.Table_3fb4e68115eb4b9f98a57da13662a443 [id=ec989ffdd24046fb905f96c691a823d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.431326 2378 tablet_bootstrap.cc:654] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:46.431852 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1807c0737d744d4cab29393e9122ff6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.433197 2372 ts_tablet_manager.cc:1950] T ec13c5386abe414ba25589e03d5b850a P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:46.434125 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5228f482c0b4fcbbdf8ebc31bb62819" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.434756 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 874) took 5995 ms. Trace:
I20251026 02:56:46.434851 2372 rpcz_store.cc:276] 1026 02:56:40.439279 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.439954 (+ 675us) service_pool.cc:225] Handling call
1026 02:56:46.413701 (+5973747us) ts_tablet_manager.cc:1151] Deleting tablet ec13c5386abe414ba25589e03d5b850a
1026 02:56:46.413713 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.418115 (+ 4402us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.422055 (+ 3940us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.434735 (+ 12680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5973586}
I20251026 02:56:46.435170 2372 tablet_replica.cc:333] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:46.435190 1975 raft_consensus.cc:3060] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.435340 2372 raft_consensus.cc:2243] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.435508 2372 raft_consensus.cc:2272] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.436067 2372 ts_tablet_manager.cc:1916] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.439122 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet ec13c5386abe414ba25589e03d5b850a (table default.Table_3792bcbbc16b4be0b113cb5f07b9261b [id=8a3d1c20763445c499bd655eb99deb56]) successfully deleted
I20251026 02:56:46.442682 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c1e63ac8c5645f3ba1578792c0a0dae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.443162 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_089d38ef94b2431f926063d1eb24d294"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.444301 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_089d38ef94b2431f926063d1eb24d294 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.452193 2372 ts_tablet_manager.cc:1929] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251026 02:56:46.452369 2372 log.cc:1199] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/340bf8c78a88413299b61a52f0a578f3
I20251026 02:56:46.452749 2372 ts_tablet_manager.cc:1950] T 340bf8c78a88413299b61a52f0a578f3 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:46.453496 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 877) took 5776 ms. Trace:
I20251026 02:56:46.453581 2372 rpcz_store.cc:276] 1026 02:56:40.677276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.677355 (+ 79us) service_pool.cc:225] Handling call
1026 02:56:46.435128 (+5757773us) ts_tablet_manager.cc:1151] Deleting tablet 340bf8c78a88413299b61a52f0a578f3
1026 02:56:46.435137 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.445702 (+ 10565us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.452066 (+ 6364us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.453474 (+ 1408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5757547}
I20251026 02:56:46.453953 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 885) took 5599 ms. Trace:
I20251026 02:56:46.454010 2372 rpcz_store.cc:276] 1026 02:56:40.855017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.855088 (+ 71us) service_pool.cc:225] Handling call
1026 02:56:46.453837 (+5598749us) ts_tablet_manager.cc:1151] Deleting tablet 572afa86823247a8b1744813350c8eff
1026 02:56:46.453846 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.453939 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5598482}
I20251026 02:56:46.454071 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 340bf8c78a88413299b61a52f0a578f3 (table default.Table_26dd741c83ff419f8c9bc808e83d9c72 [id=c162ef89d2f14e91a24a6d8c2c37c261]) successfully deleted
I20251026 02:56:46.454290 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 886) took 5586 ms. Trace:
I20251026 02:56:46.454345 2372 rpcz_store.cc:276] 1026 02:56:40.867378 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.869765 (+ 2387us) service_pool.cc:225] Handling call
1026 02:56:46.454210 (+5584445us) ts_tablet_manager.cc:1151] Deleting tablet 7c148c74fbcd4d30b9ba0ea059284427
1026 02:56:46.454217 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.454279 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5582134}
W20251026 02:56:46.454486 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 572afa86823247a8b1744813350c8eff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 572afa86823247a8b1744813350c8eff already in progress: creating tablet
I20251026 02:56:46.454596 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 887) took 5577 ms. Trace:
I20251026 02:56:46.454648 2372 rpcz_store.cc:276] 1026 02:56:40.877622 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.879772 (+ 2150us) service_pool.cc:225] Handling call
1026 02:56:46.454523 (+5574751us) ts_tablet_manager.cc:1151] Deleting tablet 83416598159a4eb598cf91a4a4e6f44a
1026 02:56:46.454530 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.454585 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5574559}
I20251026 02:56:46.454864 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 889) took 5557 ms. Trace:
W20251026 02:56:46.454833 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7c148c74fbcd4d30b9ba0ea059284427 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c148c74fbcd4d30b9ba0ea059284427 already in progress: creating tablet
I20251026 02:56:46.454914 2372 rpcz_store.cc:276] 1026 02:56:40.897078 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.897219 (+ 141us) service_pool.cc:225] Handling call
1026 02:56:46.454807 (+5557588us) ts_tablet_manager.cc:1151] Deleting tablet a378e913112c443db869016e3b98fd0e
1026 02:56:46.454814 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.454853 (+ 39us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5557286}
I20251026 02:56:46.455101 2372 tablet_replica.cc:333] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:46.455185 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 83416598159a4eb598cf91a4a4e6f44a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83416598159a4eb598cf91a4a4e6f44a already in progress: creating tablet
I20251026 02:56:46.455263 2372 raft_consensus.cc:2243] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.455399 2372 raft_consensus.cc:2272] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:46.455394 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a378e913112c443db869016e3b98fd0e because the tablet was not found. No further retry: Not found: Tablet not found: a378e913112c443db869016e3b98fd0e
I20251026 02:56:46.459383 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fb2ff87cf0248a0a4a77f2733ef867b with delete_type TABLET_DATA_DELETED (replaced by tablet ce11081bf85f44e8bf1f92ce0f4bf152) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.459911 2225 tablet_service.cc:1505] Processing CreateTablet for tablet 260c6153465e48cb9b4fd24b4f7f7a83 (DEFAULT_TABLE table=default.Table_4e1e3945b1094e179d628f897d2241dc [id=98f8a1adf5694216885ba0f52b186fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.460361 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 260c6153465e48cb9b4fd24b4f7f7a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.461985 3988 raft_consensus.cc:493] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.462132 3988 raft_consensus.cc:515] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.462723 2372 ts_tablet_manager.cc:1916] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:46.463383 2053 leader_election.cc:343] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.464082 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 86e483c52e514f1da54335479f59d68e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:46.465076 2194 leader_election.cc:343] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.465361 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 865) took 4182 ms. Trace:
I20251026 02:56:46.465503 2376 rpcz_store.cc:276] 1026 02:56:42.282453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.282500 (+ 47us) service_pool.cc:225] Handling call
1026 02:56:46.340558 (+4058058us) ts_tablet_manager.cc:1151] Deleting tablet d65b0d79e2e54af9810022b683c3288b
1026 02:56:46.340569 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.381804 (+ 41235us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.384915 (+ 3111us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.465344 (+ 80429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4057919}
I20251026 02:56:46.465889 2376 tablet_replica.cc:333] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:46.466079 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "209fa71af13142a58efb9f3dab3eecd3" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.467702 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 0c6963a62d3949e68662a35d9d528f1b (DEFAULT_TABLE table=default.Table_03342f2363b44f6599f1d4cc6fef2be8 [id=f940fc82956f473580c87805ceac50b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.469058 3997 raft_consensus.cc:493] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.469218 3997 raft_consensus.cc:515] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.469794 3997 leader_election.cc:290] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.468204 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c6963a62d3949e68662a35d9d528f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.468371 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cf76ef8f6af445eb0ce15c8d79136c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.468425 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c7efb70c0804c149f36440a4a9e6ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.476986 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 1807c0737d744d4cab29393e9122ff6b (DEFAULT_TABLE table=default.Table_3fb4e68115eb4b9f98a57da13662a443 [id=ec989ffdd24046fb905f96c691a823d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.477419 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1807c0737d744d4cab29393e9122ff6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.478048 2376 raft_consensus.cc:2243] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.478380 2376 raft_consensus.cc:2272] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.478797 2372 ts_tablet_manager.cc:1929] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:46.478921 2372 log.cc:1199] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/a012d868a78f4324a92db83c81931b9d
I20251026 02:56:46.479171 2376 ts_tablet_manager.cc:1916] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.479318 2372 ts_tablet_manager.cc:1950] T a012d868a78f4324a92db83c81931b9d P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:46.480051 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 888) took 5583 ms. Trace:
I20251026 02:56:46.480119 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 41b7b0c36ba748f0a1565892dbe98a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.480151 2372 rpcz_store.cc:276] 1026 02:56:40.896585 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.911641 (+ 15056us) service_pool.cc:225] Handling call
1026 02:56:46.455067 (+5543426us) ts_tablet_manager.cc:1151] Deleting tablet a012d868a78f4324a92db83c81931b9d
1026 02:56:46.455073 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.468854 (+ 13781us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.478689 (+ 9835us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.480031 (+ 1342us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5541271}
I20251026 02:56:46.480599 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 890) took 5573 ms. Trace:
I20251026 02:56:46.480674 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet a012d868a78f4324a92db83c81931b9d (table default.Table_3dd70def7f1046ea9f2b5bbb7a74e4cb [id=b4e3ea5575a140ac85b4c0cbd57c9ec4]) successfully deleted
I20251026 02:56:46.480681 2372 rpcz_store.cc:276] 1026 02:56:40.907252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.915093 (+ 7841us) service_pool.cc:225] Handling call
1026 02:56:46.480474 (+5565381us) ts_tablet_manager.cc:1151] Deleting tablet 9d12453c4225455881336b9f31d8e378
1026 02:56:46.480485 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.480580 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5565118}
I20251026 02:56:46.480947 2372 tablet_replica.cc:333] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:46.481144 2372 raft_consensus.cc:2243] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:46.481160 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9d12453c4225455881336b9f31d8e378 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d12453c4225455881336b9f31d8e378 already in progress: creating tablet
I20251026 02:56:46.481293 2372 raft_consensus.cc:2272] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.481824 2372 ts_tablet_manager.cc:1916] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.482153 3997 raft_consensus.cc:493] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.482295 3997 raft_consensus.cc:515] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.482987 3997 leader_election.cc:290] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:46.486256 2195 consensus_peers.cc:597] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c31d2704d1f14ddead69c794cc631ec1. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:46.486428 2195 leader_election.cc:304] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.462954 3988 leader_election.cc:290] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.487746 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet d65b0d79e2e54af9810022b683c3288b (table default.Table_bcac335a2c9a479fb5dea86b2aeffde6 [id=30a5783033cb4f7f929e766fb47d92ff]) successfully deleted
W20251026 02:56:46.488184 2053 leader_election.cc:343] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.489140 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95c9fbae62db4850b0f02a2443a3432b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.489648 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dbbfb77851d443f8cf4d798f3e07e59" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.382297 2381 tablet_bootstrap.cc:654] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:46.496503 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6a263a68d6e4b5d8d35c78595266e2b consensus state has no leader
W20251026 02:56:46.500362 2053 leader_election.cc:343] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.500975 4009 raft_consensus.cc:2804] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:46.501073 4009 raft_consensus.cc:493] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:46.501160 4009 raft_consensus.cc:3060] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.503652 2372 ts_tablet_manager.cc:1929] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:46.504530 2372 log.cc:1199] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/7f22c6c163c34ed48a0b7eeb5a574e76
I20251026 02:56:46.503883 4009 raft_consensus.cc:515] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.505101 2372 ts_tablet_manager.cc:1950] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:46.505196 4009 leader_election.cc:290] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.506307 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 892) took 5524 ms. Trace:
I20251026 02:56:46.506662 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5228f482c0b4fcbbdf8ebc31bb62819" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:46.506862 2262 raft_consensus.cc:2393] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b6b0a6c4e144982b67980118cddba48 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:46.506889 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dbbfb77851d443f8cf4d798f3e07e59" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:46.507076 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "209fa71af13142a58efb9f3dab3eecd3" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:46.507688 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95c9fbae62db4850b0f02a2443a3432b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:46.508432 2372 rpcz_store.cc:276] 1026 02:56:40.981416 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:40.983921 (+ 2505us) service_pool.cc:225] Handling call
1026 02:56:46.480905 (+5496984us) ts_tablet_manager.cc:1151] Deleting tablet 7f22c6c163c34ed48a0b7eeb5a574e76
1026 02:56:46.480913 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.493651 (+ 12738us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.503539 (+ 9888us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.506288 (+ 2749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5496661}
I20251026 02:56:46.509157 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 893) took 5496 ms. Trace:
I20251026 02:56:46.509204 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 7f22c6c163c34ed48a0b7eeb5a574e76 (table default.Table_0d7b9434a0494a618c25ceebd7f65de8 [id=bb7b13116ad94fa9bd15a1cdb15d207b]) successfully deleted
I20251026 02:56:46.509246 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5228f482c0b4fcbbdf8ebc31bb62819" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:46.509593 2053 leader_election.cc:343] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.509620 2372 rpcz_store.cc:276] 1026 02:56:41.012373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.013774 (+ 1401us) service_pool.cc:225] Handling call
1026 02:56:46.509040 (+5495266us) ts_tablet_manager.cc:1151] Deleting tablet a32b1a6b856c47b993a308db94dac357
1026 02:56:46.509050 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.509143 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5495076}
I20251026 02:56:46.510001 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 898) took 5406 ms. Trace:
W20251026 02:56:46.510092 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a32b1a6b856c47b993a308db94dac357 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a32b1a6b856c47b993a308db94dac357 already in progress: creating tablet
I20251026 02:56:46.510075 2372 rpcz_store.cc:276] 1026 02:56:41.103921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.104099 (+ 178us) service_pool.cc:225] Handling call
1026 02:56:46.509921 (+5405822us) ts_tablet_manager.cc:1151] Deleting tablet 049a915b35224cdaa38df77c734a0f01
1026 02:56:46.509930 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.509987 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5405632}
I20251026 02:56:46.510308 2372 tablet_replica.cc:333] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:46.510465 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 049a915b35224cdaa38df77c734a0f01 because the tablet was not found. No further retry: Not found: Tablet not found: 049a915b35224cdaa38df77c734a0f01
I20251026 02:56:46.510500 2372 raft_consensus.cc:2243] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.510637 2372 raft_consensus.cc:2272] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.511215 2372 ts_tablet_manager.cc:1916] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:46.512710 2194 leader_election.cc:343] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.514546 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 260c6153465e48cb9b4fd24b4f7f7a83 (DEFAULT_TABLE table=default.Table_4e1e3945b1094e179d628f897d2241dc [id=98f8a1adf5694216885ba0f52b186fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.514966 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 260c6153465e48cb9b4fd24b4f7f7a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.515748 2092 tablet_service.cc:1505] Processing CreateTablet for tablet eafb10b378e34ed3b4a33e56c6600759 (DEFAULT_TABLE table=default.Table_22ed9a263e5942e4b6fdd13731e49e1d [id=71c92396f68f4ea480392c515e534360]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.516083 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eafb10b378e34ed3b4a33e56c6600759. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.519484 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 827afd5a50c24795b6e9ccec14046e77 (DEFAULT_TABLE table=default.Table_ea91bad0b400465a98d6aec2534bcabe [id=0dd19dad498a45f6a2b052041f81e18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.519841 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827afd5a50c24795b6e9ccec14046e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.520537 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 260c6153465e48cb9b4fd24b4f7f7a83 (DEFAULT_TABLE table=default.Table_4e1e3945b1094e179d628f897d2241dc [id=98f8a1adf5694216885ba0f52b186fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.520980 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5228f482c0b4fcbbdf8ebc31bb62819" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:46.521162 2116 raft_consensus.cc:3060] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.521116 1959 tablet_service.cc:1505] Processing CreateTablet for tablet eafb10b378e34ed3b4a33e56c6600759 (DEFAULT_TABLE table=default.Table_22ed9a263e5942e4b6fdd13731e49e1d [id=71c92396f68f4ea480392c515e534360]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.521598 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 827afd5a50c24795b6e9ccec14046e77 (DEFAULT_TABLE table=default.Table_ea91bad0b400465a98d6aec2534bcabe [id=0dd19dad498a45f6a2b052041f81e18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.522039 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 9ad54b48951a4b4ba0e53530069abcec (DEFAULT_TABLE table=default.Table_b754adc158c44242939aa8d9878827e5 [id=ad1e7757ccf543a3ad62d5d8bae7bb3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.522819 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 260c6153465e48cb9b4fd24b4f7f7a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.524168 2116 raft_consensus.cc:2468] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:46.524345 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ad54b48951a4b4ba0e53530069abcec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.524856 2053 leader_election.cc:304] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.525394 3988 raft_consensus.cc:2749] T f5228f482c0b4fcbbdf8ebc31bb62819 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:46.526017 2053 leader_election.cc:343] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.526206 2053 leader_election.cc:304] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.526612 3997 raft_consensus.cc:2749] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:46.527196 2053 leader_election.cc:343] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.527362 2053 leader_election.cc:304] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.527787 3988 raft_consensus.cc:2749] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:46.527992 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827afd5a50c24795b6e9ccec14046e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.528119 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet a012d868a78f4324a92db83c81931b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:46.528617 2053 leader_election.cc:343] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.528707 2053 leader_election.cc:304] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.529743 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 51a39824f5404d45b1756a2581aa1da4
I20251026 02:56:46.531464 3988 raft_consensus.cc:2749] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:46.532078 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 51a39824f5404d45b1756a2581aa1da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:46.532348 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eafb10b378e34ed3b4a33e56c6600759. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:46.532567 2194 consensus_peers.cc:597] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:46.532764 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 827afd5a50c24795b6e9ccec14046e77 (DEFAULT_TABLE table=default.Table_ea91bad0b400465a98d6aec2534bcabe [id=0dd19dad498a45f6a2b052041f81e18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.532562 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 9ad54b48951a4b4ba0e53530069abcec (DEFAULT_TABLE table=default.Table_b754adc158c44242939aa8d9878827e5 [id=ad1e7757ccf543a3ad62d5d8bae7bb3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.536366 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 51a39824f5404d45b1756a2581aa1da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.537031 2228 tablet_service.cc:1505] Processing CreateTablet for tablet eafb10b378e34ed3b4a33e56c6600759 (DEFAULT_TABLE table=default.Table_22ed9a263e5942e4b6fdd13731e49e1d [id=71c92396f68f4ea480392c515e534360]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.539124 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dbbfb77851d443f8cf4d798f3e07e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.539235 2195 leader_election.cc:304] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.540444 4069 raft_consensus.cc:2804] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:46.540633 4069 raft_consensus.cc:697] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:46.540969 4069 consensus_queue.cc:237] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.548672 2378 tablet_bootstrap.cc:492] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:46.548827 2378 ts_tablet_manager.cc:1403] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.120s user 0.022s sys 0.017s
I20251026 02:56:46.549641 2378 raft_consensus.cc:359] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.549824 2378 raft_consensus.cc:385] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.549878 2378 raft_consensus.cc:740] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.550093 2378 consensus_queue.cc:260] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.551051 2375 tablet_bootstrap.cc:492] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:46.551240 2375 ts_tablet_manager.cc:1403] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.302s user 0.006s sys 0.002s
I20251026 02:56:46.552034 2375 raft_consensus.cc:359] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.552246 2375 raft_consensus.cc:385] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.552338 2375 raft_consensus.cc:740] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.552603 2375 consensus_queue.cc:260] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:46.552987 2053 consensus_peers.cc:597] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:46.553143 2375 ts_tablet_manager.cc:1434] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:46.553431 2375 tablet_bootstrap.cc:492] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:46.555424 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 51a39824f5404d45b1756a2581aa1da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.555671 2378 ts_tablet_manager.cc:1434] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251026 02:56:46.556564 2372 ts_tablet_manager.cc:1929] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:46.556782 2372 log.cc:1199] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/8c1e63ac8c5645f3ba1578792c0a0dae
I20251026 02:56:46.557178 2372 ts_tablet_manager.cc:1950] T 8c1e63ac8c5645f3ba1578792c0a0dae P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:46.558058 2375 tablet_bootstrap.cc:654] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:46.558473 2378 tablet_bootstrap.cc:492] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:46.560727 2378 tablet_bootstrap.cc:654] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:46.567586 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1406c22ee4ee4f94acecad9546208e79
I20251026 02:56:46.572676 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 899) took 5468 ms. Trace:
I20251026 02:56:46.572937 2372 rpcz_store.cc:276] 1026 02:56:41.104049 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.104328 (+ 279us) service_pool.cc:225] Handling call
1026 02:56:46.510264 (+5405936us) ts_tablet_manager.cc:1151] Deleting tablet 8c1e63ac8c5645f3ba1578792c0a0dae
1026 02:56:46.510272 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.544483 (+ 34211us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.548057 (+ 3574us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.572650 (+ 24593us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5405800}
I20251026 02:56:46.573572 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 8c1e63ac8c5645f3ba1578792c0a0dae (table default.Table_528af198a403496a8c1b27fa5821123b [id=e6aed02773494e3c8694deedf311adc6]) successfully deleted
I20251026 02:56:46.573611 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (table default.Table_30611e161c274e8b99bd1251e5e527aa [id=b5d91d3e8a2643c09fee85976972391a]). Expected version 1 got 0
I20251026 02:56:46.574088 2372 tablet_replica.cc:333] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:46.574381 2372 raft_consensus.cc:2243] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.574661 2372 raft_consensus.cc:2272] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.575023 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 1406c22ee4ee4f94acecad9546208e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.575713 3988 raft_consensus.cc:493] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.575955 3988 raft_consensus.cc:515] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.575740 2372 ts_tablet_manager.cc:1916] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.576766 3988 leader_election.cc:290] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.580099 2381 tablet_bootstrap.cc:492] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:46.581233 2381 ts_tablet_manager.cc:1403] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.246s user 0.002s sys 0.002s
I20251026 02:56:46.582859 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99d318aa0d1847c58167bf0af9e75aa2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.583464 4067 raft_consensus.cc:493] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.583700 4067 raft_consensus.cc:515] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.584537 4067 leader_election.cc:290] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:46.585129 2053 leader_election.cc:343] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.586032 2381 raft_consensus.cc:359] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.587122 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet 1406c22ee4ee4f94acecad9546208e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:46.589344 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48105e54d6324622a032e1288a46188c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:46.589933 2194 leader_election.cc:343] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.590646 2381 raft_consensus.cc:385] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.592258 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 9ad54b48951a4b4ba0e53530069abcec (DEFAULT_TABLE table=default.Table_b754adc158c44242939aa8d9878827e5 [id=ad1e7757ccf543a3ad62d5d8bae7bb3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.592406 2381 raft_consensus.cc:740] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.592738 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ad54b48951a4b4ba0e53530069abcec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.593353 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48105e54d6324622a032e1288a46188c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:46.593549 2116 raft_consensus.cc:2468] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:46.596849 2381 consensus_queue.cc:260] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.597222 2378 tablet_bootstrap.cc:492] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:46.597389 2378 ts_tablet_manager.cc:1403] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20251026 02:56:46.597420 2381 ts_tablet_manager.cc:1434] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251026 02:56:46.597708 2381 tablet_bootstrap.cc:492] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:46.599356 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet 34c6a7c774044411a4733b115080c4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:46.599596 2195 leader_election.cc:304] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.599947 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d0ccee4f250840c49e212c09b25cb0d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:46.600524 2376 ts_tablet_manager.cc:1929] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251026 02:56:46.600651 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0ccee4f250840c49e212c09b25cb0d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.600617 4068 raft_consensus.cc:2804] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:46.600740 4068 raft_consensus.cc:493] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:46.600811 4068 raft_consensus.cc:3060] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.600709 2376 log.cc:1199] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/6108219254bd448c852096c0c967384b
I20251026 02:56:46.601414 2376 ts_tablet_manager.cc:1950] T 6108219254bd448c852096c0c967384b P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:46.602136 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_dbd53db2b8684c5ba304639f97a47556"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.602698 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbd53db2b8684c5ba304639f97a47556 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.602905 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 866) took 4172 ms. Trace:
I20251026 02:56:46.603691 4068 raft_consensus.cc:515] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.601608 2378 raft_consensus.cc:359] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.604918 2378 raft_consensus.cc:385] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.605005 2378 raft_consensus.cc:740] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.604852 2376 rpcz_store.cc:276] 1026 02:56:42.430252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.430327 (+ 75us) service_pool.cc:225] Handling call
1026 02:56:46.465848 (+4035521us) ts_tablet_manager.cc:1151] Deleting tablet 6108219254bd448c852096c0c967384b
1026 02:56:46.465857 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.496066 (+ 30209us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.600370 (+104304us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.602879 (+ 2509us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4003803}
I20251026 02:56:46.605254 2378 consensus_queue.cc:260] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.605506 2376 tablet_replica.cc:333] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:46.605789 2378 ts_tablet_manager.cc:1434] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251026 02:56:46.606060 2378 tablet_bootstrap.cc:492] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:46.606038 2376 raft_consensus.cc:2243] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.606266 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 6108219254bd448c852096c0c967384b (table default.Table_5af1941b0082444c8bd348a080af20ee [id=6342ae7b8acf4137a16ebf978ce7e15b]) successfully deleted
I20251026 02:56:46.606464 2376 raft_consensus.cc:2272] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.607003 2376 ts_tablet_manager.cc:1916] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.613997 4068 leader_election.cc:290] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.614889 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99d318aa0d1847c58167bf0af9e75aa2" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:46.615922 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 53a1aaf540294ea9876b640087bd2dea (DEFAULT_TABLE table=default.Table_4836734d4c9f4d37b4f223c1147dbc30 [id=9c3ca2a7350743aa9fa2c61f590c4c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.616883 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48105e54d6324622a032e1288a46188c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:46.618774 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48105e54d6324622a032e1288a46188c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:46.619159 2109 raft_consensus.cc:3060] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.608481 2378 tablet_bootstrap.cc:654] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:46.621438 2194 leader_election.cc:343] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.622817 4069 raft_consensus.cc:493] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.623044 4069 raft_consensus.cc:515] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.623970 4069 leader_election.cc:290] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.624686 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "386e1bd4635b4a7e8e7f249b577fd978" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.624425 2109 raft_consensus.cc:2468] T 48105e54d6324622a032e1288a46188c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:46.625906 2053 leader_election.cc:343] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.626202 2053 leader_election.cc:304] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.626834 3988 raft_consensus.cc:2749] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:46.628072 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4c255180d55843908eaf691ea0b0a6ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.628717 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c255180d55843908eaf691ea0b0a6ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.628945 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 53a1aaf540294ea9876b640087bd2dea (DEFAULT_TABLE table=default.Table_4836734d4c9f4d37b4f223c1147dbc30 [id=9c3ca2a7350743aa9fa2c61f590c4c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.629477 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53a1aaf540294ea9876b640087bd2dea. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:46.630728 2194 leader_election.cc:343] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.631999 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "386e1bd4635b4a7e8e7f249b577fd978" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:46.633417 2225 tablet_service.cc:1505] Processing CreateTablet for tablet f48b61a1fe5e45fb8e614426b58ba254 (DEFAULT_TABLE table=default.Table_439dbab03ccd44c7869b2f12fb6431c5 [id=4cc433189bc145afa185011ece707e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.633518 2233 tablet_service.cc:1505] Processing CreateTablet for tablet ee40dd09a4aa4300a594e6ba72dc9fbc (DEFAULT_TABLE table=default.Table_1c36ab4645984dc4b2280de23324521b [id=022005dc887a40148ce4999b8cd1ebe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.633668 2109 raft_consensus.cc:2468] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:46.634012 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 9db78aec81114b0593ca8d452865a265 (DEFAULT_TABLE table=default.Table_d87f21f83121454ab8069f24d8b147ea [id=8d1fefe46e1a4c089bc9fc3f6b0f577c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.634842 2195 leader_election.cc:304] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.635066 4067 raft_consensus.cc:2804] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:46.635166 4067 raft_consensus.cc:697] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:46.635399 4067 consensus_queue.cc:237] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.636765 2195 leader_election.cc:304] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.637116 4068 consensus_queue.cc:1048] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:46.637487 4068 raft_consensus.cc:2804] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251026 02:56:46.638089 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e91466733c4142a7b2879334c0b67aaa consensus state has no leader
I20251026 02:56:46.638288 1955 tablet_service.cc:1505] Processing CreateTablet for tablet ee40dd09a4aa4300a594e6ba72dc9fbc (DEFAULT_TABLE table=default.Table_1c36ab4645984dc4b2280de23324521b [id=022005dc887a40148ce4999b8cd1ebe9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:46.638401 1649 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 662f2f0dfc804ddbbc4b71118a71d048 (table default.Table_49ed105063f845bd92ee37a1ce8d97a7 [id=8fb2c3acbda445068bbf1706e7e8e49d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 662f2f0dfc804ddbbc4b71118a71d048 consensus state has no leader
I20251026 02:56:46.638660 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee40dd09a4aa4300a594e6ba72dc9fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.640684 4068 raft_consensus.cc:493] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:46.640802 4068 raft_consensus.cc:3060] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.643527 4068 raft_consensus.cc:515] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.644402 4068 leader_election.cc:290] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.644798 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 9db78aec81114b0593ca8d452865a265 (DEFAULT_TABLE table=default.Table_d87f21f83121454ab8069f24d8b147ea [id=8d1fefe46e1a4c089bc9fc3f6b0f577c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.645257 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db78aec81114b0593ca8d452865a265. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.648434 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "386e1bd4635b4a7e8e7f249b577fd978" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:46.648643 2116 raft_consensus.cc:3060] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.651602 2116 raft_consensus.cc:2468] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:46.653843 2195 leader_election.cc:304] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:46.653836 1959 tablet_service.cc:1505] Processing CreateTablet for tablet f48b61a1fe5e45fb8e614426b58ba254 (DEFAULT_TABLE table=default.Table_439dbab03ccd44c7869b2f12fb6431c5 [id=4cc433189bc145afa185011ece707e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.653941 3984 consensus_queue.cc:1048] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:46.654259 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f48b61a1fe5e45fb8e614426b58ba254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.654903 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 1406c22ee4ee4f94acecad9546208e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.655732 4069 raft_consensus.cc:2804] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:46.655867 4069 raft_consensus.cc:697] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:46.656122 4069 consensus_queue.cc:237] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.660494 2372 ts_tablet_manager.cc:1929] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:46.661245 2372 log.cc:1199] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/86e483c52e514f1da54335479f59d68e
I20251026 02:56:46.662969 2372 ts_tablet_manager.cc:1950] T 86e483c52e514f1da54335479f59d68e P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:46.666234 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "386e1bd4635b4a7e8e7f249b577fd978" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:46.668092 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_431d446f4c7b4a7b85f4386e28da2d3b" } modify_external_catalogs: true
I20251026 02:56:46.668358 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_431d446f4c7b4a7b85f4386e28da2d3b" } modify_external_catalogs: true
W20251026 02:56:46.671711 2194 leader_election.cc:343] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.672477 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 900) took 5533 ms. Trace:
I20251026 02:56:46.672912 2372 rpcz_store.cc:276] 1026 02:56:41.138616 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.151528 (+ 12912us) service_pool.cc:225] Handling call
1026 02:56:46.574030 (+5422502us) ts_tablet_manager.cc:1151] Deleting tablet 86e483c52e514f1da54335479f59d68e
1026 02:56:46.574042 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.645151 (+ 71109us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.660061 (+ 14910us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.672340 (+ 12279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5421859}
I20251026 02:56:46.675035 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 903) took 5406 ms. Trace:
I20251026 02:56:46.675836 2372 rpcz_store.cc:276] 1026 02:56:41.268856 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.269023 (+ 167us) service_pool.cc:225] Handling call
1026 02:56:46.674353 (+5405330us) ts_tablet_manager.cc:1151] Deleting tablet 94bdaa97e7604040b7fc94fe22e743bf
1026 02:56:46.674473 (+ 120us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.674905 (+ 432us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5369493}
I20251026 02:56:46.677624 2372 tablet_replica.cc:333] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:46.678566 2372 raft_consensus.cc:2243] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.681370 2378 tablet_bootstrap.cc:492] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:46.681316 2082 tablet_service.cc:1505] Processing CreateTablet for tablet ee40dd09a4aa4300a594e6ba72dc9fbc (DEFAULT_TABLE table=default.Table_1c36ab4645984dc4b2280de23324521b [id=022005dc887a40148ce4999b8cd1ebe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.681493 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 86e483c52e514f1da54335479f59d68e (table default.Table_6d91747dd6164f0d84509497a93aeeea [id=f33faf56d22346bdaa270b22e673315a]) successfully deleted
I20251026 02:56:46.681612 2378 ts_tablet_manager.cc:1403] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.076s user 0.000s sys 0.003s
I20251026 02:56:46.681846 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee40dd09a4aa4300a594e6ba72dc9fbc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:46.681823 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 94bdaa97e7604040b7fc94fe22e743bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94bdaa97e7604040b7fc94fe22e743bf already in progress: creating tablet
I20251026 02:56:46.682394 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 9db78aec81114b0593ca8d452865a265 (DEFAULT_TABLE table=default.Table_d87f21f83121454ab8069f24d8b147ea [id=8d1fefe46e1a4c089bc9fc3f6b0f577c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.682363 2378 raft_consensus.cc:359] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.682530 2378 raft_consensus.cc:385] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.682576 2378 raft_consensus.cc:740] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.682747 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db78aec81114b0593ca8d452865a265. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.682930 2378 consensus_queue.cc:260] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.688582 2096 tablet_service.cc:1505] Processing CreateTablet for tablet f48b61a1fe5e45fb8e614426b58ba254 (DEFAULT_TABLE table=default.Table_439dbab03ccd44c7869b2f12fb6431c5 [id=4cc433189bc145afa185011ece707e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.688832 2378 ts_tablet_manager.cc:1434] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20251026 02:56:46.689034 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f48b61a1fe5e45fb8e614426b58ba254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.694553 2372 raft_consensus.cc:2272] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.697203 2372 ts_tablet_manager.cc:1916] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.702309 1955 tablet_service.cc:1505] Processing CreateTablet for tablet a8b28be2419d44c994468ebe0c5472b4 (DEFAULT_TABLE table=default.Table_fe71a1f386b943db95b6684602e55c22 [id=65b1f8aea3eb4df1be3276eccfc8b931]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.711244 2094 tablet_service.cc:1505] Processing CreateTablet for tablet a8b28be2419d44c994468ebe0c5472b4 (DEFAULT_TABLE table=default.Table_fe71a1f386b943db95b6684602e55c22 [id=65b1f8aea3eb4df1be3276eccfc8b931]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.711791 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8b28be2419d44c994468ebe0c5472b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.713519 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9f9c766d51344e4683974de3d9d367e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:46.713940 2231 tablet_service.cc:1505] Processing CreateTablet for tablet aca91661c2b54b79917fcf00f87fe6f9 (DEFAULT_TABLE table=default.Table_c7ef2f8329744217bb25610a6afb4889 [id=d71bb7c3b19648c38eddb19f2cea20f9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:46.714156 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f9c766d51344e4683974de3d9d367e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.715001 2082 tablet_service.cc:1505] Processing CreateTablet for tablet aca91661c2b54b79917fcf00f87fe6f9 (DEFAULT_TABLE table=default.Table_c7ef2f8329744217bb25610a6afb4889 [id=d71bb7c3b19648c38eddb19f2cea20f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.715998 2234 tablet_service.cc:1505] Processing CreateTablet for tablet a8b28be2419d44c994468ebe0c5472b4 (DEFAULT_TABLE table=default.Table_fe71a1f386b943db95b6684602e55c22 [id=65b1f8aea3eb4df1be3276eccfc8b931]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.700368 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 1eabfc95caa84afc9a8956a7ac13fea6 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b8e1d79d2a247dba25540daf6b81bf0) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.722648 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aca91661c2b54b79917fcf00f87fe6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.728789 2098 tablet_service.cc:1505] Processing CreateTablet for tablet f7858b6a20814d5cb494be2a5001ae49 (DEFAULT_TABLE table=default.Table_d4fb1ce9057f4cde849a4a8050bf6a18 [id=47eba24166a44e7a83c6b6e44b0bfc4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.734915 2092 tablet_service.cc:1505] Processing CreateTablet for tablet c34406d36c6e48dc973584bd653a9e42 (DEFAULT_TABLE table=default.Table_39b5d99fe09d427389ea14393b4f56ef [id=390e5f55c3d149129bcfd5a38f7e1b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.741180 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34406d36c6e48dc973584bd653a9e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.751194 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7858b6a20814d5cb494be2a5001ae49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.752370 2227 tablet_service.cc:1505] Processing CreateTablet for tablet f7858b6a20814d5cb494be2a5001ae49 (DEFAULT_TABLE table=default.Table_d4fb1ce9057f4cde849a4a8050bf6a18 [id=47eba24166a44e7a83c6b6e44b0bfc4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.753477 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8b28be2419d44c994468ebe0c5472b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.757606 1974 raft_consensus.cc:3060] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.758060 1944 tablet_service.cc:1505] Processing CreateTablet for tablet aca91661c2b54b79917fcf00f87fe6f9 (DEFAULT_TABLE table=default.Table_c7ef2f8329744217bb25610a6afb4889 [id=d71bb7c3b19648c38eddb19f2cea20f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.758126 2229 tablet_service.cc:1505] Processing CreateTablet for tablet c34406d36c6e48dc973584bd653a9e42 (DEFAULT_TABLE table=default.Table_39b5d99fe09d427389ea14393b4f56ef [id=390e5f55c3d149129bcfd5a38f7e1b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.758598 1956 tablet_service.cc:1505] Processing CreateTablet for tablet f7858b6a20814d5cb494be2a5001ae49 (DEFAULT_TABLE table=default.Table_d4fb1ce9057f4cde849a4a8050bf6a18 [id=47eba24166a44e7a83c6b6e44b0bfc4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.758663 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 8b6e4376477a4f29913fe215a3901c53 (DEFAULT_TABLE table=default.Table_f44743d6a730429da0c9d4900b298c9b [id=7211e7f9bb124d559b8afd972f409f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.759639 3988 raft_consensus.cc:493] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.759778 3988 raft_consensus.cc:515] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:46.760064 2194 consensus_peers.cc:597] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 352a284902c94f088e2e6ae006162db0. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:46.760422 3988 leader_election.cc:290] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.760409 1960 tablet_service.cc:1505] Processing CreateTablet for tablet c34406d36c6e48dc973584bd653a9e42 (DEFAULT_TABLE table=default.Table_39b5d99fe09d427389ea14393b4f56ef [id=390e5f55c3d149129bcfd5a38f7e1b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.761647 2376 ts_tablet_manager.cc:1929] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:46.761788 2376 log.cc:1199] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/6a3491b303f94fe2a2c838297f73d464
I20251026 02:56:46.761651 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7858b6a20814d5cb494be2a5001ae49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.762696 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aca91661c2b54b79917fcf00f87fe6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.763918 2378 tablet_bootstrap.cc:492] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:46.764827 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 8b6e4376477a4f29913fe215a3901c53 (DEFAULT_TABLE table=default.Table_f44743d6a730429da0c9d4900b298c9b [id=7211e7f9bb124d559b8afd972f409f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.766989 2375 tablet_bootstrap.cc:492] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:46.767107 2375 ts_tablet_manager.cc:1403] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.214s user 0.005s sys 0.000s
I20251026 02:56:46.767706 2375 raft_consensus.cc:359] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.768138 2375 raft_consensus.cc:385] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.768442 2375 raft_consensus.cc:740] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.768724 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfac07917ee34e6dba3fac5e081f6b43" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:46.769568 2053 leader_election.cc:343] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.769291 2375 consensus_queue.cc:260] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.770947 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfac07917ee34e6dba3fac5e081f6b43" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.773088 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 53a1aaf540294ea9876b640087bd2dea (DEFAULT_TABLE table=default.Table_4836734d4c9f4d37b4f223c1147dbc30 [id=9c3ca2a7350743aa9fa2c61f590c4c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.774252 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53a1aaf540294ea9876b640087bd2dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.775211 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6e4376477a4f29913fe215a3901c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.775441 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34406d36c6e48dc973584bd653a9e42. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:46.775839 2053 leader_election.cc:343] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.775957 2053 leader_election.cc:304] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.776201 3988 raft_consensus.cc:2749] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:46.779712 2372 ts_tablet_manager.cc:1929] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:46.780654 2372 log.cc:1199] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/352a284902c94f088e2e6ae006162db0
I20251026 02:56:46.782786 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 8b6e4376477a4f29913fe215a3901c53 (DEFAULT_TABLE table=default.Table_f44743d6a730429da0c9d4900b298c9b [id=7211e7f9bb124d559b8afd972f409f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.784188 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6e4376477a4f29913fe215a3901c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.784790 2376 ts_tablet_manager.cc:1950] T 6a3491b303f94fe2a2c838297f73d464 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:46.785148 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 867) took 4354 ms. Trace:
I20251026 02:56:46.785233 2376 rpcz_store.cc:276] 1026 02:56:42.430380 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.430427 (+ 47us) service_pool.cc:225] Handling call
1026 02:56:46.605451 (+4175024us) ts_tablet_manager.cc:1151] Deleting tablet 6a3491b303f94fe2a2c838297f73d464
1026 02:56:46.605463 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.619736 (+ 14273us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.761537 (+141801us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.785126 (+ 23589us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4143285}
I20251026 02:56:46.785972 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 6a3491b303f94fe2a2c838297f73d464 (table default.Table_30b96d2d4369458390803c6be18efe0e [id=9fd6e480fc3543149959773e691a3ddd]) successfully deleted
I20251026 02:56:46.786700 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 869) took 4234 ms. Trace:
I20251026 02:56:46.786795 2376 rpcz_store.cc:276] 1026 02:56:42.552741 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.554989 (+ 2248us) service_pool.cc:225] Handling call
1026 02:56:46.785532 (+4230543us) ts_tablet_manager.cc:1151] Deleting tablet 5b8e1d79d2a247dba25540daf6b81bf0
1026 02:56:46.785541 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.786675 (+ 1134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4214294}
I20251026 02:56:46.787168 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 870) took 4229 ms. Trace:
I20251026 02:56:46.787238 2376 rpcz_store.cc:276] 1026 02:56:42.557451 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.557525 (+ 74us) service_pool.cc:225] Handling call
1026 02:56:46.787062 (+4229537us) ts_tablet_manager.cc:1151] Deleting tablet 2f197526e7d34f849dbb4f28d8d414cf
1026 02:56:46.787073 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.787154 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4215761}
W20251026 02:56:46.787999 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 5b8e1d79d2a247dba25540daf6b81bf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b8e1d79d2a247dba25540daf6b81bf0 already in progress: creating tablet
W20251026 02:56:46.788343 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 2f197526e7d34f849dbb4f28d8d414cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f197526e7d34f849dbb4f28d8d414cf already in progress: creating tablet
I20251026 02:56:46.795075 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6b5b0a005339470aa86ae92151ee1920" } new_table_name: "default.Table_309ea91c2d5f4843ba3f8b91cbca53a4" modify_external_catalogs: true
I20251026 02:56:46.796078 1657 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 4274877b37114119b4513e4a2a30b8df
I20251026 02:56:46.796458 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_759cf0037bcd4bf58341bd2f01711cdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.797058 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_759cf0037bcd4bf58341bd2f01711cdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.797973 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 873) took 4189 ms. Trace:
I20251026 02:56:46.798099 2376 rpcz_store.cc:276] 1026 02:56:42.608653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.608720 (+ 67us) service_pool.cc:225] Handling call
1026 02:56:46.787454 (+4178734us) ts_tablet_manager.cc:1151] Deleting tablet 8977f824ff374d05b4ed9d88c67329f9
1026 02:56:46.787465 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.797946 (+ 10481us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4178494}
W20251026 02:56:46.802611 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 8977f824ff374d05b4ed9d88c67329f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8977f824ff374d05b4ed9d88c67329f9 already in progress: creating tablet
I20251026 02:56:46.802457 2372 ts_tablet_manager.cc:1950] T 352a284902c94f088e2e6ae006162db0 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:46.805598 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 874) took 4192 ms. Trace:
I20251026 02:56:46.805711 2376 rpcz_store.cc:276] 1026 02:56:42.612789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.612862 (+ 73us) service_pool.cc:225] Handling call
1026 02:56:46.798417 (+4185555us) ts_tablet_manager.cc:1151] Deleting tablet a4f4d7f87916425999574fd5b9f2fdf7
1026 02:56:46.798427 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.805570 (+ 7143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4185331}
I20251026 02:56:46.806453 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 875) took 4189 ms. Trace:
I20251026 02:56:46.806933 2376 rpcz_store.cc:276] 1026 02:56:42.617286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.617363 (+ 77us) service_pool.cc:225] Handling call
1026 02:56:46.806211 (+4188848us) ts_tablet_manager.cc:1151] Deleting tablet ac82c14e7dcb448b845d998249e2b339
1026 02:56:46.806221 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.806434 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4188593}
W20251026 02:56:46.807719 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet ac82c14e7dcb448b845d998249e2b339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac82c14e7dcb448b845d998249e2b339 already in progress: creating tablet
W20251026 02:56:46.808738 2053 consensus_peers.cc:597] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:46.808912 1916 consensus_peers.cc:597] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d212170f056a44dfb15cfcb7f9fbae2f. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:46.811872 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 872) took 4203 ms. Trace:
I20251026 02:56:46.812665 2376 rpcz_store.cc:276] 1026 02:56:42.608480 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.620574 (+ 12094us) service_pool.cc:225] Handling call
1026 02:56:46.808062 (+4187488us) ts_tablet_manager.cc:1151] Deleting tablet 663076e362c44837beb322aa965cde42
1026 02:56:46.808073 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.811847 (+ 3774us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4187211}
W20251026 02:56:46.813901 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 663076e362c44837beb322aa965cde42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 663076e362c44837beb322aa965cde42 already in progress: creating tablet
I20251026 02:56:46.814792 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 876) took 4188 ms. Trace:
I20251026 02:56:46.815021 2376 rpcz_store.cc:276] 1026 02:56:42.626694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.626769 (+ 75us) service_pool.cc:225] Handling call
1026 02:56:46.814508 (+4187739us) ts_tablet_manager.cc:1151] Deleting tablet b8fe7825fa8d46669772cf84bc4b3afc
1026 02:56:46.814520 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.814735 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4187437}
I20251026 02:56:46.814854 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 904) took 5545 ms. Trace:
W20251026 02:56:46.815901 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b8fe7825fa8d46669772cf84bc4b3afc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8fe7825fa8d46669772cf84bc4b3afc already in progress: creating tablet
I20251026 02:56:46.815584 2372 rpcz_store.cc:276] 1026 02:56:41.268967 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.269656 (+ 689us) service_pool.cc:225] Handling call
1026 02:56:46.677338 (+5407682us) ts_tablet_manager.cc:1151] Deleting tablet 352a284902c94f088e2e6ae006162db0
1026 02:56:46.677351 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.731776 (+ 54425us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.778950 (+ 47174us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.814637 (+ 35687us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5372317}
I20251026 02:56:46.818114 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 352a284902c94f088e2e6ae006162db0 (table default.Table_6cd4ee82e9ec406f971669b7f0722677 [id=414375003cd44b5c8c9f0249b339c28a]) successfully deleted
I20251026 02:56:46.820609 2376 tablet_replica.cc:333] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:46.820924 3988 consensus_queue.cc:1048] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:46.821380 2376 raft_consensus.cc:2243] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.821853 2376 raft_consensus.cc:2272] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.812264 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 5cab1d6e8d8048f791ec23b78d276848 (DEFAULT_TABLE table=default.Table_7ba3891cfe0b46329d460a7a3881d8d0 [id=3cd62b627c1d4b5d875e4bf0a9158aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.823809 2376 ts_tablet_manager.cc:1916] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.828939 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cab1d6e8d8048f791ec23b78d276848. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.829416 2375 ts_tablet_manager.cc:1434] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.062s user 0.000s sys 0.003s
I20251026 02:56:46.833302 2375 tablet_bootstrap.cc:492] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:46.835698 2195 consensus_peers.cc:597] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 352a284902c94f088e2e6ae006162db0. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:46.837738 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 914) took 5350 ms. Trace:
I20251026 02:56:46.840832 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 68ad3d4a2d484badb4b35884517b72f1 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:46.843429 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet bc4ca4423e3a4e0c96846babec931fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.844102 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 5cab1d6e8d8048f791ec23b78d276848 (DEFAULT_TABLE table=default.Table_7ba3891cfe0b46329d460a7a3881d8d0 [id=3cd62b627c1d4b5d875e4bf0a9158aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.844633 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cab1d6e8d8048f791ec23b78d276848. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:46.840565 2372 rpcz_store.cc:276] 1026 02:56:41.487251 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.489723 (+ 2472us) service_pool.cc:225] Handling call
1026 02:56:46.818667 (+5328944us) ts_tablet_manager.cc:1151] Deleting tablet 3aed4584b49a4eccac2ba44454866d49
1026 02:56:46.818681 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.837312 (+ 18631us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5328352}
W20251026 02:56:46.850049 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3aed4584b49a4eccac2ba44454866d49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3aed4584b49a4eccac2ba44454866d49 already in progress: creating tablet
I20251026 02:56:46.853632 2372 tablet_replica.cc:333] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:46.861930 2372 raft_consensus.cc:2243] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:46.864766 2372 raft_consensus.cc:2272] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:46.872975 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 3bcd7dec9c2149069e48dcf5e564595e
I20251026 02:56:46.874928 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bcd7dec9c2149069e48dcf5e564595e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:46.875175 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bcd7dec9c2149069e48dcf5e564595e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:46.873601 2372 ts_tablet_manager.cc:1916] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:46.879103 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_857acd1afff5412b8945e40b98c1dde6" } new_table_name: "default.Table_791ef632f7b24cf2a51f8de05d259ad5" modify_external_catalogs: true
I20251026 02:56:46.879756 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 5cab1d6e8d8048f791ec23b78d276848 (DEFAULT_TABLE table=default.Table_7ba3891cfe0b46329d460a7a3881d8d0 [id=3cd62b627c1d4b5d875e4bf0a9158aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.881311 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9b0b596ec6014ba288a0cb8110dfc2a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:46.882005 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b0b596ec6014ba288a0cb8110dfc2a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:46.890683 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bcd7dec9c2149069e48dcf5e564595e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:46.902655 3988 raft_consensus.cc:493] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.902827 3988 raft_consensus.cc:515] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.903460 3988 leader_election.cc:290] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.903884 2241 tablet_service.cc:1505] Processing CreateTablet for tablet c79d33c3005548f1a22f7acc6e4097ab (DEFAULT_TABLE table=default.Table_9e61f50aaa744075aa0488e84f1166b5 [id=f1e0b0c9ad994b6aa08256387d28be11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:46.904953 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a3f3fc3c7a04d05bae28472251f3acc" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.905447 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a3f3fc3c7a04d05bae28472251f3acc" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:46.905987 2053 leader_election.cc:343] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:46.906251 2053 leader_election.cc:343] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.906335 2053 leader_election.cc:304] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.906859 3997 raft_consensus.cc:2749] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:46.911645 3997 consensus_queue.cc:1048] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:46.916596 2378 tablet_bootstrap.cc:654] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:46.918367 4009 raft_consensus.cc:493] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.918526 4009 raft_consensus.cc:515] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.919132 4009 leader_election.cc:290] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:46.919776 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95c9fbae62db4850b0f02a2443a3432b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:46.919978 2109 raft_consensus.cc:2468] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:46.920631 2195 leader_election.cc:304] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:46.924185 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95c9fbae62db4850b0f02a2443a3432b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:46.924870 2194 leader_election.cc:343] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.932427 4069 raft_consensus.cc:2804] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:46.932355 1659 catalog_manager.cc:5649] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:46.932651 4069 raft_consensus.cc:493] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:46.932837 1659 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 386e1bd4635b4a7e8e7f249b577fd978 (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]). Expected version 2 got 0
I20251026 02:56:46.933022 4069 raft_consensus.cc:3060] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.933161 1659 catalog_manager.cc:5649] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:46.957123 3988 raft_consensus.cc:493] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.957492 3988 raft_consensus.cc:515] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.956930 2375 tablet_bootstrap.cc:654] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:46.958209 3988 leader_election.cc:290] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.959086 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d43b6f22dd4f4f9c9657f21514550312" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:46.959602 2053 leader_election.cc:343] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.961576 3988 raft_consensus.cc:493] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.961717 3988 raft_consensus.cc:515] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.962343 3988 leader_election.cc:290] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.963053 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41b7b0c36ba748f0a1565892dbe98a78" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.963495 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41b7b0c36ba748f0a1565892dbe98a78" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:46.963677 2256 raft_consensus.cc:2468] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
W20251026 02:56:46.964149 2053 leader_election.cc:343] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.964394 2053 leader_election.cc:304] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.964813 3997 raft_consensus.cc:2804] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:46.964887 3997 raft_consensus.cc:493] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:46.964938 3997 raft_consensus.cc:3060] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.964896 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d43b6f22dd4f4f9c9657f21514550312" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.967792 3997 raft_consensus.cc:515] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:46.969362 2053 leader_election.cc:343] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.969492 2053 leader_election.cc:304] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.969913 3988 raft_consensus.cc:2749] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:46.970814 3997 leader_election.cc:290] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.971571 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41b7b0c36ba748f0a1565892dbe98a78" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:46.971647 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41b7b0c36ba748f0a1565892dbe98a78" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:46.971834 2256 raft_consensus.cc:3060] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:46.974962 2256 raft_consensus.cc:2468] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:46.975476 2053 leader_election.cc:304] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:46.975734 3988 raft_consensus.cc:493] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.975862 3988 raft_consensus.cc:515] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.976429 3988 leader_election.cc:290] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.976563 3988 raft_consensus.cc:2804] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:46.976656 3988 raft_consensus.cc:697] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:46.976899 3988 consensus_queue.cc:237] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:46.978634 2053 leader_election.cc:343] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.979346 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a00bc04b3654f90af342986751344e6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.979842 2378 tablet_bootstrap.cc:492] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
W20251026 02:56:46.980016 2053 leader_election.cc:343] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.980351 2378 ts_tablet_manager.cc:1403] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.286s user 0.006s sys 0.002s
I20251026 02:56:46.980446 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a00bc04b3654f90af342986751344e6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:46.981035 2053 leader_election.cc:343] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:46.981137 2053 leader_election.cc:304] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.981534 3997 raft_consensus.cc:2749] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:46.984140 2378 raft_consensus.cc:359] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.985020 2378 raft_consensus.cc:385] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:46.985298 2378 raft_consensus.cc:740] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:46.986820 2378 consensus_queue.cc:260] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.989681 3997 raft_consensus.cc:493] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:46.989843 3997 raft_consensus.cc:515] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.990572 2372 ts_tablet_manager.cc:1929] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:46.991979 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:46.992197 2256 raft_consensus.cc:2468] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:46.991204 3997 leader_election.cc:290] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:46.992681 2053 leader_election.cc:304] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:46.993263 4067 raft_consensus.cc:493] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:46.993402 4067 raft_consensus.cc:515] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:46.994338 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1ca5b8379c94deab1130bfe8bdc87b0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.995080 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:46.995359 1974 raft_consensus.cc:2468] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:46.996320 4067 leader_election.cc:290] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:46.996544 4088 log.cc:927] Time spent T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
W20251026 02:56:46.996937 2195 leader_election.cc:343] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: b1ca5b8379c94deab1130bfe8bdc87b0
I20251026 02:56:46.997036 2195 leader_election.cc:304] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:46.992887 2372 log.cc:1199] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/dba102838945453cbf44e1cfe505a52c
I20251026 02:56:47.000149 4009 raft_consensus.cc:2749] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:46.992996 3997 raft_consensus.cc:2804] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 02:56:47.000293 3997 raft_consensus.cc:493] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:47.000414 3997 raft_consensus.cc:3060] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:47.001688 2378 ts_tablet_manager.cc:1434] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20251026 02:56:47.007411 3997 raft_consensus.cc:515] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.010156 2372 ts_tablet_manager.cc:1950] T dba102838945453cbf44e1cfe505a52c P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:47.013350 2378 tablet_bootstrap.cc:492] T 48105e54d6324622a032e1288a46188c P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:47.013746 2375 tablet_bootstrap.cc:492] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:47.017015 2375 ts_tablet_manager.cc:1403] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.184s user 0.008s sys 0.000s
I20251026 02:56:47.022450 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 915) took 5406 ms. Trace:
I20251026 02:56:47.025039 4089 raft_consensus.cc:493] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:47.025203 4089 raft_consensus.cc:515] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.025861 4089 leader_election.cc:290] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.024164 2372 rpcz_store.cc:276] 1026 02:56:41.616144 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.616201 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:46.852357 (+5236156us) ts_tablet_manager.cc:1151] Deleting tablet dba102838945453cbf44e1cfe505a52c
1026 02:56:46.852593 (+ 236us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.930252 (+ 77659us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:46.988498 (+ 58246us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.021972 (+ 33474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5235433}
I20251026 02:56:47.027320 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1ca5b8379c94deab1130bfe8bdc87b0" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:47.027487 1916 leader_election.cc:343] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: b1ca5b8379c94deab1130bfe8bdc87b0
I20251026 02:56:47.027523 2259 raft_consensus.cc:2468] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:47.028030 1915 leader_election.cc:304] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:47.028541 4089 raft_consensus.cc:2804] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 02:56:47.028618 4089 raft_consensus.cc:493] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:47.028679 4089 raft_consensus.cc:3060] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:47.032490 4089 raft_consensus.cc:515] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.033187 2238 tablet_service.cc:1505] Processing CreateTablet for tablet 599a88ec0e76409c97ab62d6f55ebcd0 (DEFAULT_TABLE table=default.Table_6bf9cedc1f4c46479c8d1b407d1fdc5d [id=8a5342d19b2347b182762d84dd03a104]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:47.034731 1916 leader_election.cc:343] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: b1ca5b8379c94deab1130bfe8bdc87b0
I20251026 02:56:47.035439 4089 leader_election.cc:290] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.036278 1960 tablet_service.cc:1505] Processing CreateTablet for tablet c79d33c3005548f1a22f7acc6e4097ab (DEFAULT_TABLE table=default.Table_9e61f50aaa744075aa0488e84f1166b5 [id=f1e0b0c9ad994b6aa08256387d28be11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.036638 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c79d33c3005548f1a22f7acc6e4097ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.038681 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 599a88ec0e76409c97ab62d6f55ebcd0 (DEFAULT_TABLE table=default.Table_6bf9cedc1f4c46479c8d1b407d1fdc5d [id=8a5342d19b2347b182762d84dd03a104]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.039268 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1ca5b8379c94deab1130bfe8bdc87b0" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:47.039449 2259 raft_consensus.cc:3060] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:47.039503 4069 raft_consensus.cc:515] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.039922 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 599a88ec0e76409c97ab62d6f55ebcd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.039990 4069 leader_election.cc:290] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:47.042665 2259 raft_consensus.cc:2468] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 2.
I20251026 02:56:47.043932 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95c9fbae62db4850b0f02a2443a3432b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:47.044124 2116 raft_consensus.cc:3060] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.046994 2116 raft_consensus.cc:2468] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:47.048333 2195 leader_election.cc:304] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:47.048482 1915 leader_election.cc:304] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:47.048662 4067 raft_consensus.cc:2804] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:47.048964 4067 raft_consensus.cc:697] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:47.049286 4067 consensus_queue.cc:237] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.051934 2226 tablet_service.cc:1505] Processing CreateTablet for tablet fd9eef082c084267b10163fee196ba6f (DEFAULT_TABLE table=default.Table_04a58bdc201c45eaafcdbcfc8ccf93bd [id=b926a666f79b4eddbc59ec883e675bc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.055296 2243 tablet_service.cc:1505] Processing CreateTablet for tablet d8148110a8774ca8a7824c0461c65332 (DEFAULT_TABLE table=default.Table_88c37cad8c6841d69e4ed80515b92b5e [id=25d2dc1090554faebe721704b8988977]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.056291 4095 raft_consensus.cc:2804] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 02:56:47.056607 4095 raft_consensus.cc:697] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:47.056978 4095 consensus_queue.cc:237] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.060354 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95c9fbae62db4850b0f02a2443a3432b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:47.060873 1944 tablet_service.cc:1505] Processing CreateTablet for tablet fd9eef082c084267b10163fee196ba6f (DEFAULT_TABLE table=default.Table_04a58bdc201c45eaafcdbcfc8ccf93bd [id=b926a666f79b4eddbc59ec883e675bc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.061286 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd9eef082c084267b10163fee196ba6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.060683 2375 raft_consensus.cc:359] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:47.064694 2194 leader_election.cc:343] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.066100 2375 raft_consensus.cc:385] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.073870 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet f80b5c4c4e2d4c14a8a5292a05127bc1
I20251026 02:56:47.074796 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet f80b5c4c4e2d4c14a8a5292a05127bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:47.074816 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet f80b5c4c4e2d4c14a8a5292a05127bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:47.077548 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_914472564539427caff89590b14d9d01" } new_table_name: "default.Table_be8c60cde79445d79d373c6fa4c0f745" modify_external_catalogs: true
I20251026 02:56:47.068537 2375 raft_consensus.cc:740] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.080188 2376 ts_tablet_manager.cc:1929] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:47.080531 2376 log.cc:1199] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/a6f9ae327aa4401baf66a737eaa93cf6
I20251026 02:56:47.081583 2376 ts_tablet_manager.cc:1950] T a6f9ae327aa4401baf66a737eaa93cf6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:47.084422 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 916) took 5442 ms. Trace:
I20251026 02:56:47.085070 2372 rpcz_store.cc:276] 1026 02:56:41.642300 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.642372 (+ 72us) service_pool.cc:225] Handling call
1026 02:56:47.081563 (+5439191us) ts_tablet_manager.cc:1151] Deleting tablet 887dc91626664f23b39cf70a6bb9381e
1026 02:56:47.081639 (+ 76us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.084261 (+ 2622us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5438845}
I20251026 02:56:47.083595 2375 consensus_queue.cc:260] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.087457 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 917) took 5420 ms. Trace:
I20251026 02:56:47.087845 2372 rpcz_store.cc:276] 1026 02:56:41.666549 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.666599 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:47.086766 (+5420167us) ts_tablet_manager.cc:1151] Deleting tablet ade0f7de597049ffb45274d60c9c0932
1026 02:56:47.086858 (+ 92us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.087379 (+ 521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5419916}
I20251026 02:56:47.089536 3997 leader_election.cc:290] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.092571 2378 tablet_bootstrap.cc:654] T 48105e54d6324622a032e1288a46188c P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.092692 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 69885e58f9e04979b6157c61e4c0f261 (DEFAULT_TABLE table=default.Table_1821574ebc7f452b9cc0d8231f78cbdc [id=216da41d45b24046b44678c1ff34c17e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.093221 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69885e58f9e04979b6157c61e4c0f261. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.097944 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:47.098230 2259 raft_consensus.cc:3060] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:47.100539 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fd8f02126b24285bbfa575540ba0f09" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:47.101099 2259 raft_consensus.cc:2468] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 2.
I20251026 02:56:47.101162 1974 raft_consensus.cc:3060] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:47.105343 4097 raft_consensus.cc:493] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.106343 4097 raft_consensus.cc:515] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.110908 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c17caa9e7ba04ae2a7d056d8fa619a10" } new_table_name: "default.Table_9c224d9efebb4e9ea1f3b2b2bb63335e" modify_external_catalogs: true
I20251026 02:56:47.105733 1960 tablet_service.cc:1505] Processing CreateTablet for tablet d8148110a8774ca8a7824c0461c65332 (DEFAULT_TABLE table=default.Table_88c37cad8c6841d69e4ed80515b92b5e [id=25d2dc1090554faebe721704b8988977]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.114033 4098 consensus_queue.cc:1048] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 02:56:47.117323 2053 leader_election.cc:304] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:47.121229 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 69885e58f9e04979b6157c61e4c0f261 (DEFAULT_TABLE table=default.Table_1821574ebc7f452b9cc0d8231f78cbdc [id=216da41d45b24046b44678c1ff34c17e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.121685 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8148110a8774ca8a7824c0461c65332. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.125061 2375 ts_tablet_manager.cc:1434] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.104s user 0.006s sys 0.002s
I20251026 02:56:47.129441 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet dba102838945453cbf44e1cfe505a52c (table default.Table_8ffce6cf6b2d44aa8a9459e834de1d80 [id=92310ee99b5c428c9773b0f7a5462a4d]) successfully deleted
W20251026 02:56:47.129758 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 887dc91626664f23b39cf70a6bb9381e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 887dc91626664f23b39cf70a6bb9381e already in progress: creating tablet
W20251026 02:56:47.130089 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ade0f7de597049ffb45274d60c9c0932 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ade0f7de597049ffb45274d60c9c0932 already in progress: creating tablet
I20251026 02:56:47.136046 2375 tablet_bootstrap.cc:492] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:47.138027 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet f80b5c4c4e2d4c14a8a5292a05127bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:46 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:47.138362 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 69885e58f9e04979b6157c61e4c0f261 (DEFAULT_TABLE table=default.Table_1821574ebc7f452b9cc0d8231f78cbdc [id=216da41d45b24046b44678c1ff34c17e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.138314 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 599a88ec0e76409c97ab62d6f55ebcd0 (DEFAULT_TABLE table=default.Table_6bf9cedc1f4c46479c8d1b407d1fdc5d [id=8a5342d19b2347b182762d84dd03a104]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.138408 2082 tablet_service.cc:1505] Processing CreateTablet for tablet d8148110a8774ca8a7824c0461c65332 (DEFAULT_TABLE table=default.Table_88c37cad8c6841d69e4ed80515b92b5e [id=25d2dc1090554faebe721704b8988977]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.139293 2096 tablet_service.cc:1505] Processing CreateTablet for tablet c79d33c3005548f1a22f7acc6e4097ab (DEFAULT_TABLE table=default.Table_9e61f50aaa744075aa0488e84f1166b5 [id=f1e0b0c9ad994b6aa08256387d28be11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.139626 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8148110a8774ca8a7824c0461c65332. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.139878 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c79d33c3005548f1a22f7acc6e4097ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.140497 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69885e58f9e04979b6157c61e4c0f261. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.127065 4098 raft_consensus.cc:2804] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 02:56:47.145787 4098 raft_consensus.cc:697] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [term 2 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:47.147413 1659 catalog_manager.cc:5649] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:47.138747 2100 tablet_service.cc:1505] Processing CreateTablet for tablet fd9eef082c084267b10163fee196ba6f (DEFAULT_TABLE table=default.Table_04a58bdc201c45eaafcdbcfc8ccf93bd [id=b926a666f79b4eddbc59ec883e675bc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.151307 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 918) took 5433 ms. Trace:
I20251026 02:56:47.152175 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd9eef082c084267b10163fee196ba6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.153312 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 599a88ec0e76409c97ab62d6f55ebcd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.158882 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 893) took 4064 ms. Trace:
I20251026 02:56:47.159284 2376 rpcz_store.cc:276] 1026 02:56:43.094087 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.094144 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:46.820474 (+3726330us) ts_tablet_manager.cc:1151] Deleting tablet a6f9ae327aa4401baf66a737eaa93cf6
1026 02:56:46.820487 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.002717 (+182230us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.080079 (+ 77362us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.158789 (+ 78710us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3726086}
I20251026 02:56:47.152192 2372 rpcz_store.cc:276] 1026 02:56:41.717757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.717810 (+ 53us) service_pool.cc:225] Handling call
1026 02:56:47.128180 (+5410370us) ts_tablet_manager.cc:1151] Deleting tablet ecbed9dd56e94cad8d7387155b934c50
1026 02:56:47.150495 (+ 22315us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.151161 (+ 666us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5407684}
W20251026 02:56:47.163539 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ecbed9dd56e94cad8d7387155b934c50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecbed9dd56e94cad8d7387155b934c50 already in progress: creating tablet
I20251026 02:56:47.164520 2372 tablet_replica.cc:333] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:47.166141 2372 raft_consensus.cc:2243] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:47.168094 4097 leader_election.cc:290] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.170763 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0bacd15b41149b5856d20068711b432" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:47.173285 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 906) took 3917 ms. Trace:
I20251026 02:56:47.173715 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0bacd15b41149b5856d20068711b432" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:47.174010 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet a6f9ae327aa4401baf66a737eaa93cf6 (table default.Table_828c16a96a47489f835a891f34d707b1 [id=432234d273ca4c9ea53c7a078c274637]) successfully deleted
I20251026 02:56:47.174408 2376 rpcz_store.cc:276] 1026 02:56:43.255298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.255346 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:47.161104 (+3905758us) ts_tablet_manager.cc:1151] Deleting tablet 9b9dc0825e0844f68d5afcc201db737c
1026 02:56:47.161163 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.173159 (+ 11996us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3905518}
W20251026 02:56:47.176306 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 9b9dc0825e0844f68d5afcc201db737c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b9dc0825e0844f68d5afcc201db737c already in progress: creating tablet
I20251026 02:56:47.176354 1975 raft_consensus.cc:3060] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.177130 2376 tablet_replica.cc:333] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:47.177815 2376 raft_consensus.cc:2243] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:47.178412 2376 raft_consensus.cc:2272] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:47.180343 2376 ts_tablet_manager.cc:1916] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:47.188699 1982 raft_consensus.cc:3060] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.203171 4089 raft_consensus.cc:493] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.203331 4089 raft_consensus.cc:515] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.203984 4089 leader_election.cc:290] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:47.205241 1916 leader_election.cc:343] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: f719c720cc474181a63786a3e248ab2c
I20251026 02:56:47.205791 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f719c720cc474181a63786a3e248ab2c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:47.206007 2259 raft_consensus.cc:2468] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:47.195152 1974 raft_consensus.cc:2468] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 2.
I20251026 02:56:47.206401 1915 leader_election.cc:304] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:47.148710 4098 consensus_queue.cc:237] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.206815 4089 raft_consensus.cc:2804] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:47.206903 4089 raft_consensus.cc:493] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:47.206980 4089 raft_consensus.cc:3060] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.209919 4089 raft_consensus.cc:515] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:47.211336 1916 leader_election.cc:343] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: f719c720cc474181a63786a3e248ab2c
I20251026 02:56:47.212809 4089 leader_election.cc:290] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.212992 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f719c720cc474181a63786a3e248ab2c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:47.213176 2259 raft_consensus.cc:3060] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:47.212399 2053 leader_election.cc:343] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:47.220324 2053 leader_election.cc:343] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.221144 2053 leader_election.cc:304] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:47.222097 4097 raft_consensus.cc:2749] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:47.216145 2259 raft_consensus.cc:2468] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:47.223744 1915 leader_election.cc:304] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:47.224249 4089 raft_consensus.cc:2804] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:47.224392 4089 raft_consensus.cc:697] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:47.224680 4089 consensus_queue.cc:237] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.230695 4009 consensus_queue.cc:1048] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:47.234831 2195 consensus_peers.cc:597] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3dbe8002df6c4b64b8ec8260ca6b9cf5. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:47.237663 3984 raft_consensus.cc:493] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.237848 3984 raft_consensus.cc:515] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.238579 3984 leader_election.cc:290] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.239441 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c6a7c774044411a4733b115080c4c6" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:47.239135 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c6a7c774044411a4733b115080c4c6" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:47.242332 1915 leader_election.cc:304] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:47.242784 4095 raft_consensus.cc:2749] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:47.243777 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827afd5a50c24795b6e9ccec14046e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.244323 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c79d33c3005548f1a22f7acc6e4097ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.244679 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34406d36c6e48dc973584bd653a9e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.247771 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8b28be2419d44c994468ebe0c5472b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.248306 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6e4376477a4f29913fe215a3901c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.248476 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eafb10b378e34ed3b4a33e56c6600759. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.248659 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee40dd09a4aa4300a594e6ba72dc9fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.248818 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7858b6a20814d5cb494be2a5001ae49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.249197 2243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8148110a8774ca8a7824c0461c65332. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.249554 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd9eef082c084267b10163fee196ba6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.250577 2381 tablet_bootstrap.cc:654] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.251525 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cab1d6e8d8048f791ec23b78d276848. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.252307 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69885e58f9e04979b6157c61e4c0f261. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.254689 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53a1aaf540294ea9876b640087bd2dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.256312 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aca91661c2b54b79917fcf00f87fe6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.259020 2377 ts_tablet_manager.cc:1929] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:47.259222 2377 log.cc:1199] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/882fe81da21344ed99ea735fa79f73c0
I20251026 02:56:47.260321 2225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f48b61a1fe5e45fb8e614426b58ba254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.261844 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db78aec81114b0593ca8d452865a265. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.264350 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 599a88ec0e76409c97ab62d6f55ebcd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.266002 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ad54b48951a4b4ba0e53530069abcec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.270126 2377 ts_tablet_manager.cc:1950] T 882fe81da21344ed99ea735fa79f73c0 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:47.271157 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 891) took 6168 ms. Trace:
I20251026 02:56:47.271355 2377 rpcz_store.cc:276] 1026 02:56:41.102932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.103423 (+ 491us) service_pool.cc:225] Handling call
1026 02:56:46.293682 (+5190259us) ts_tablet_manager.cc:1151] Deleting tablet 882fe81da21344ed99ea735fa79f73c0
1026 02:56:46.293693 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:46.518670 (+224977us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.258937 (+740267us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.271132 (+ 12195us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5190085}
I20251026 02:56:47.271963 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 882fe81da21344ed99ea735fa79f73c0 (table default.Table_18e8723148dd46e6a56e5716a15d01bc [id=156e8ddc92b1471e8dce064bcb23306b]) successfully deleted
I20251026 02:56:47.272854 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 896) took 5987 ms. Trace:
I20251026 02:56:47.272994 2377 rpcz_store.cc:276] 1026 02:56:41.285704 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.285771 (+ 67us) service_pool.cc:225] Handling call
1026 02:56:47.272718 (+5986947us) ts_tablet_manager.cc:1151] Deleting tablet 94bdaa97e7604040b7fc94fe22e743bf
1026 02:56:47.272730 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.272836 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5986744}
I20251026 02:56:47.278088 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0d7b9434a0494a618c25ceebd7f65de8" } modify_external_catalogs: true
I20251026 02:56:47.278311 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_0d7b9434a0494a618c25ceebd7f65de8" } modify_external_catalogs: true
W20251026 02:56:47.282907 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f383f638e63040379bf749b8774fd7ed consensus state has no leader
I20251026 02:56:47.287086 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6fc619ce52194bd58287fde601a7a24d" } modify_external_catalogs: true
I20251026 02:56:47.287284 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6fc619ce52194bd58287fde601a7a24d" } modify_external_catalogs: true
W20251026 02:56:47.289131 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 94bdaa97e7604040b7fc94fe22e743bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94bdaa97e7604040b7fc94fe22e743bf already in progress: creating tablet
W20251026 02:56:47.292479 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 69585.5us
I20251026 02:56:47.293478 2377 tablet_replica.cc:333] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:47.293788 2377 raft_consensus.cc:2243] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:47.294226 2377 raft_consensus.cc:2272] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:47.294849 2377 ts_tablet_manager.cc:1916] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:47.294886 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 68ad3d4a2d484badb4b35884517b72f1
I20251026 02:56:47.295423 1654 catalog_manager.cc:6719] ReplaceTablet: tablet 68ad3d4a2d484badb4b35884517b72f1 deleted and replaced by tablet a0b9626c8ed14636acdf0a286a709625
I20251026 02:56:47.299450 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 68ad3d4a2d484badb4b35884517b72f1 with delete_type TABLET_DATA_DELETED (replaced by tablet a0b9626c8ed14636acdf0a286a709625) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:47.299991 4097 raft_consensus.cc:493] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.300266 4097 raft_consensus.cc:515] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.300889 4097 leader_election.cc:290] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.300851 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_53792bda07c940909d190b494a1e663b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:47.301442 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53792bda07c940909d190b494a1e663b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.302775 2375 tablet_bootstrap.cc:654] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.303406 4097 raft_consensus.cc:493] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.303598 4097 raft_consensus.cc:515] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.304173 4097 leader_election.cc:290] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.238376 2372 raft_consensus.cc:2272] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:47.305883 2053 consensus_peers.cc:597] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:47.308735 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6699b50d87964f6a89c365171253ad25" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:47.310323 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13acb028b89e47eeadff91fe61cbebf3 consensus state has no leader
I20251026 02:56:47.302651 2381 tablet_bootstrap.cc:492] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:47.311332 2381 ts_tablet_manager.cc:1403] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.714s user 0.022s sys 0.014s
I20251026 02:56:47.311986 2378 tablet_bootstrap.cc:492] T 48105e54d6324622a032e1288a46188c P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:47.312119 2378 ts_tablet_manager.cc:1403] T 48105e54d6324622a032e1288a46188c P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.300s user 0.009s sys 0.002s
I20251026 02:56:47.312081 2381 raft_consensus.cc:359] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.312270 2381 raft_consensus.cc:385] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.312330 2381 raft_consensus.cc:740] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.306849 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6699b50d87964f6a89c365171253ad25" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:47.307911 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc4ca4423e3a4e0c96846babec931fec" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:47.312687 2227 tablet_service.cc:1552] Processing DeleteTablet for tablet 68ad3d4a2d484badb4b35884517b72f1 with delete_type TABLET_DATA_DELETED (replaced by tablet a0b9626c8ed14636acdf0a286a709625) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:47.312882 2378 raft_consensus.cc:359] T 48105e54d6324622a032e1288a46188c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.313086 2378 raft_consensus.cc:385] T 48105e54d6324622a032e1288a46188c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.313141 2378 raft_consensus.cc:740] T 48105e54d6324622a032e1288a46188c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.313357 2378 consensus_queue.cc:260] T 48105e54d6324622a032e1288a46188c P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.312891 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc4ca4423e3a4e0c96846babec931fec" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:47.313647 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_56366720e640479392e1652794ea3847" } new_table_name: "default.Table_6bd4eede457348b0bd4dfe92f0ff146e" modify_external_catalogs: true
I20251026 02:56:47.313942 2378 ts_tablet_manager.cc:1434] T 48105e54d6324622a032e1288a46188c P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:47.314203 2378 tablet_bootstrap.cc:492] T 7abcdac4a70b48ad9fbba1bc8a60283b P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:47.315785 2378 tablet_bootstrap.cc:654] T 7abcdac4a70b48ad9fbba1bc8a60283b P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.317236 2372 ts_tablet_manager.cc:1916] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:47.318578 2053 leader_election.cc:343] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:47.327430 2195 consensus_peers.cc:597] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c1e63ac8c5645f3ba1578792c0a0dae. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:47.327414 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7549c6a144774b478402b14065800d6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:47.327982 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7549c6a144774b478402b14065800d6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.330216 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 68ad3d4a2d484badb4b35884517b72f1 with delete_type TABLET_DATA_DELETED (replaced by tablet a0b9626c8ed14636acdf0a286a709625) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:47.332388 2053 leader_election.cc:343] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:47.332577 2053 leader_election.cc:343] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.332646 2053 leader_election.cc:304] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:47.332880 3997 raft_consensus.cc:2749] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:47.333046 2053 leader_election.cc:343] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.333130 2053 leader_election.cc:304] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:47.333325 3997 raft_consensus.cc:2749] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:47.333518 2381 consensus_queue.cc:260] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:47.333750 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11f4c327caa644d89098496bc324b976 consensus state has no leader
I20251026 02:56:47.334275 2381 ts_tablet_manager.cc:1434] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251026 02:56:47.334524 2381 tablet_bootstrap.cc:492] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:47.334653 2376 ts_tablet_manager.cc:1929] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:47.334758 2376 log.cc:1199] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/7801f073d5d94076af9299492f86fad2
I20251026 02:56:47.335099 2376 ts_tablet_manager.cc:1950] T 7801f073d5d94076af9299492f86fad2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:47.335990 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 897) took 4114 ms. Trace:
I20251026 02:56:47.336613 2381 tablet_bootstrap.cc:654] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.337278 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a08b6588e1414d00874ff1ec6c01c00b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:47.337841 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a08b6588e1414d00874ff1ec6c01c00b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.340739 4067 consensus_queue.cc:1048] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:47.345371 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 109073eb664d4090b7a2b786146e13fa consensus state has no leader
I20251026 02:56:47.349367 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_30e98b21e65d443097e2fc01f96f4780" } modify_external_catalogs: true
I20251026 02:56:47.349551 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_30e98b21e65d443097e2fc01f96f4780" } modify_external_catalogs: true
I20251026 02:56:47.357513 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 97aa7f0266854c0a8f17e6bceac0c9b5 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:47.362673 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b161497cfeea47febcaaabb994fd8862" } modify_external_catalogs: true
W20251026 02:56:47.363533 2195 consensus_peers.cc:597] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec13c5386abe414ba25589e03d5b850a. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:47.362655 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_debeb38e035a435fa936ab479b12d30e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:47.363006 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b161497cfeea47febcaaabb994fd8862" } modify_external_catalogs: true
W20251026 02:56:47.364540 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_debeb38e035a435fa936ab479b12d30e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.371181 1657 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet a322ec2fa36448e3b66a97d50d4392a1
I20251026 02:56:47.372722 2377 ts_tablet_manager.cc:1929] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:47.372948 2377 log.cc:1199] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/352a284902c94f088e2e6ae006162db0
I20251026 02:56:47.373397 2377 ts_tablet_manager.cc:1950] T 352a284902c94f088e2e6ae006162db0 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:47.384927 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 4db275efdd344817b11a2168675cf95e (table default.Table_c30cfa52b6524323bda478613d74178b [id=1fd74186ea10498599435dccd7330d25]). Expected version 1 got 0
I20251026 02:56:47.389082 2376 rpcz_store.cc:276] 1026 02:56:43.221696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.221994 (+ 298us) service_pool.cc:225] Handling call
1026 02:56:47.176987 (+3954993us) ts_tablet_manager.cc:1151] Deleting tablet 7801f073d5d94076af9299492f86fad2
1026 02:56:47.176997 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.279277 (+102280us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.334542 (+ 55265us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.335971 (+ 1429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3921344}
I20251026 02:56:47.389699 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 7801f073d5d94076af9299492f86fad2 (table default.Table_03a9f618938a45aab881c91a801cc32c [id=12b41b47b5d243188eb73bd83a83bbdc]) successfully deleted
I20251026 02:56:47.390897 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 904) took 4138 ms. Trace:
I20251026 02:56:47.391067 2376 rpcz_store.cc:276] 1026 02:56:43.252434 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.252494 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:47.390781 (+4138287us) ts_tablet_manager.cc:1151] Deleting tablet 0245af5a2dcb4f569b62307965895c1f
1026 02:56:47.390792 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.390881 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4126403}
W20251026 02:56:47.391611 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0245af5a2dcb4f569b62307965895c1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0245af5a2dcb4f569b62307965895c1f already in progress: creating tablet
I20251026 02:56:47.350575 2378 tablet_bootstrap.cc:492] T 7abcdac4a70b48ad9fbba1bc8a60283b P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:47.392078 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 905) took 4139 ms. Trace:
I20251026 02:56:47.380900 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c1e63ac8c5645f3ba1578792c0a0dae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:47.380921 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet f719c720cc474181a63786a3e248ab2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:47.392199 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0aabfe11dedd454f8221fb5de25a268b
I20251026 02:56:47.392396 2376 rpcz_store.cc:276] 1026 02:56:43.252922 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.252976 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:47.391911 (+4138935us) ts_tablet_manager.cc:1151] Deleting tablet 868dbbfce40c4dd6b36cc220966b1986
1026 02:56:47.391922 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.392013 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4123607}
I20251026 02:56:47.392782 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 896) took 4171 ms. Trace:
I20251026 02:56:47.392844 2376 rpcz_store.cc:276] 1026 02:56:43.220958 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.222169 (+ 1211us) service_pool.cc:225] Handling call
1026 02:56:47.392693 (+4170524us) ts_tablet_manager.cc:1151] Deleting tablet 19918cb3b7d0450f9de17026f8ea3295
1026 02:56:47.392702 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.392768 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4124141}
I20251026 02:56:47.393437 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 909) took 4080 ms. Trace:
I20251026 02:56:47.393507 2376 rpcz_store.cc:276] 1026 02:56:43.313070 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.313143 (+ 73us) service_pool.cc:225] Handling call
1026 02:56:47.393354 (+4080211us) ts_tablet_manager.cc:1151] Deleting tablet 035724949fe142ea93cb998f8fdc4344
1026 02:56:47.393362 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.393424 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4074701}
I20251026 02:56:47.393805 2376 tablet_replica.cc:333] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:47.393971 2376 raft_consensus.cc:2243] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:47.394119 2376 raft_consensus.cc:2272] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:47.394629 2376 ts_tablet_manager.cc:1916] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:47.396031 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 868dbbfce40c4dd6b36cc220966b1986 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 868dbbfce40c4dd6b36cc220966b1986 already in progress: creating tablet
I20251026 02:56:47.324036 2372 ts_tablet_manager.cc:1929] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:47.325218 2375 tablet_bootstrap.cc:492] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
W20251026 02:56:47.396288 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 19918cb3b7d0450f9de17026f8ea3295 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19918cb3b7d0450f9de17026f8ea3295 already in progress: creating tablet
I20251026 02:56:47.396363 2375 ts_tablet_manager.cc:1403] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.264s user 0.008s sys 0.001s
I20251026 02:56:47.396422 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 0aabfe11dedd454f8221fb5de25a268b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:47.396472 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 035724949fe142ea93cb998f8fdc4344 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 035724949fe142ea93cb998f8fdc4344 already in progress: creating tablet
I20251026 02:56:47.392160 2378 ts_tablet_manager.cc:1403] T 7abcdac4a70b48ad9fbba1bc8a60283b P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.000s
I20251026 02:56:47.397293 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 0aabfe11dedd454f8221fb5de25a268b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:47.397396 2378 raft_consensus.cc:359] T 7abcdac4a70b48ad9fbba1bc8a60283b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.397120 2375 raft_consensus.cc:359] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.397564 2378 raft_consensus.cc:385] T 7abcdac4a70b48ad9fbba1bc8a60283b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.397619 2378 raft_consensus.cc:740] T 7abcdac4a70b48ad9fbba1bc8a60283b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.397775 2375 raft_consensus.cc:385] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.397805 2378 consensus_queue.cc:260] T 7abcdac4a70b48ad9fbba1bc8a60283b P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.397879 2375 raft_consensus.cc:740] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.398272 2378 ts_tablet_manager.cc:1434] T 7abcdac4a70b48ad9fbba1bc8a60283b P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 02:56:47.398555 2378 tablet_bootstrap.cc:492] T 386e1bd4635b4a7e8e7f249b577fd978 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:47.398294 2375 consensus_queue.cc:260] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.399860 2375 ts_tablet_manager.cc:1434] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251026 02:56:47.401342 2375 tablet_bootstrap.cc:492] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:47.361078 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet a012d868a78f4324a92db83c81931b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:47.403553 2375 tablet_bootstrap.cc:654] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.405332 2372 log.cc:1199] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/9c7efb70c0804c149f36440a4a9e6ef8
I20251026 02:56:47.405884 2372 ts_tablet_manager.cc:1950] T 9c7efb70c0804c149f36440a4a9e6ef8 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:47.406944 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 897) took 6121 ms. Trace:
I20251026 02:56:47.382112 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet 95c9fbae62db4850b0f02a2443a3432b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:47.388594 2194 consensus_peers.cc:597] T f5228f482c0b4fcbbdf8ebc31bb62819 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:47.391930 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet d43b6f22dd4f4f9c9657f21514550312 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:47.396255 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 0aabfe11dedd454f8221fb5de25a268b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:47.400959 2381 tablet_bootstrap.cc:492] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:47.407692 2381 ts_tablet_manager.cc:1403] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.000s
I20251026 02:56:47.407966 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_066e72649e794e878a6512b6dea2f868"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:47.408392 2381 raft_consensus.cc:359] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.408536 2381 raft_consensus.cc:385] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.408576 2381 raft_consensus.cc:740] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
W20251026 02:56:47.408574 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_066e72649e794e878a6512b6dea2f868 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.408774 2381 consensus_queue.cc:260] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.409220 2381 ts_tablet_manager.cc:1434] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 02:56:47.409518 2381 tablet_bootstrap.cc:492] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:47.411278 2381 tablet_bootstrap.cc:654] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.412129 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d04da838fd9f42feab6cb9d6178160a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:47.412698 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d04da838fd9f42feab6cb9d6178160a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.416415 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 921) took 5131 ms. Trace:
I20251026 02:56:47.416574 2372 rpcz_store.cc:276] 1026 02:56:42.285258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.285492 (+ 234us) service_pool.cc:225] Handling call
1026 02:56:47.163942 (+4878450us) ts_tablet_manager.cc:1151] Deleting tablet 9c7efb70c0804c149f36440a4a9e6ef8
1026 02:56:47.164099 (+ 157us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.320990 (+156891us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.323935 (+ 2945us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.416387 (+ 92452us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4861007}
I20251026 02:56:47.407034 2377 rpcz_store.cc:276] 1026 02:56:41.285833 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.286014 (+ 181us) service_pool.cc:225] Handling call
1026 02:56:47.292166 (+6006152us) ts_tablet_manager.cc:1151] Deleting tablet 352a284902c94f088e2e6ae006162db0
1026 02:56:47.292175 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.343263 (+ 51088us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.372606 (+ 29343us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.406918 (+ 34312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6005997}
W20251026 02:56:47.417128 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51bfa49903d34ff18fb5a0bb9b75295a consensus state has no leader
I20251026 02:56:47.417734 4098 raft_consensus.cc:493] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.417860 4098 raft_consensus.cc:515] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.417966 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 905) took 5899 ms. Trace:
I20251026 02:56:47.418008 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 9c7efb70c0804c149f36440a4a9e6ef8 (table default.Table_e99391c4c0f74eb3ac3b62cedada8612 [id=4397550364664316a6f1e96fb17d9509]) successfully deleted
I20251026 02:56:47.418053 2377 rpcz_store.cc:276] 1026 02:56:41.518121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.519962 (+ 1841us) service_pool.cc:225] Handling call
1026 02:56:47.417578 (+5897616us) ts_tablet_manager.cc:1151] Deleting tablet 3aed4584b49a4eccac2ba44454866d49
1026 02:56:47.417851 (+ 273us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.417951 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5897010}
I20251026 02:56:47.418457 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 352a284902c94f088e2e6ae006162db0 (table default.Table_6cd4ee82e9ec406f971669b7f0722677 [id=414375003cd44b5c8c9f0249b339c28a]) successfully deleted
I20251026 02:56:47.418474 4098 leader_election.cc:290] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:47.418689 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3aed4584b49a4eccac2ba44454866d49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3aed4584b49a4eccac2ba44454866d49 already in progress: creating tablet
I20251026 02:56:47.418689 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 910) took 5678 ms. Trace:
I20251026 02:56:47.418764 2377 rpcz_store.cc:276] 1026 02:56:41.740003 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.740059 (+ 56us) service_pool.cc:225] Handling call
1026 02:56:47.418587 (+5678528us) ts_tablet_manager.cc:1151] Deleting tablet ecbed9dd56e94cad8d7387155b934c50
1026 02:56:47.418596 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.418673 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5678332}
I20251026 02:56:47.419037 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 909) took 5679 ms. Trace:
I20251026 02:56:47.419098 2377 rpcz_store.cc:276] 1026 02:56:41.739935 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.740526 (+ 591us) service_pool.cc:225] Handling call
1026 02:56:47.418947 (+5678421us) ts_tablet_manager.cc:1151] Deleting tablet ade0f7de597049ffb45274d60c9c0932
1026 02:56:47.418955 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.419022 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5678266}
I20251026 02:56:47.419339 2377 tablet_replica.cc:333] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:47.419397 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eabfc95caa84afc9a8956a7ac13fea6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:47.419498 2377 raft_consensus.cc:2243] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 2 LEADER]: Raft consensus shutting down.
I20251026 02:56:47.419761 2377 pending_rounds.cc:70] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:47.419800 2377 pending_rounds.cc:77] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7214888576204488704 op_type: NO_OP noop_request { }
I20251026 02:56:47.419876 2377 raft_consensus.cc:2889] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:47.419946 2377 raft_consensus.cc:2272] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:47.420921 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 922) took 5135 ms. Trace:
I20251026 02:56:47.421031 2372 rpcz_store.cc:276] 1026 02:56:42.285364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.285562 (+ 198us) service_pool.cc:225] Handling call
1026 02:56:47.416971 (+5131409us) ts_tablet_manager.cc:1151] Deleting tablet ae95654d129441e08fcb3f3b355ffc22
1026 02:56:47.420818 (+ 3847us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.420900 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5114048}
W20251026 02:56:47.421505 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ae95654d129441e08fcb3f3b355ffc22 because the tablet was not found. No further retry: Not found: Tablet not found: ae95654d129441e08fcb3f3b355ffc22
W20251026 02:56:47.421516 2053 leader_election.cc:343] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.421804 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eabfc95caa84afc9a8956a7ac13fea6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:47.422271 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ecbed9dd56e94cad8d7387155b934c50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecbed9dd56e94cad8d7387155b934c50 already in progress: creating tablet
W20251026 02:56:47.422330 2053 leader_election.cc:343] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.422432 2053 leader_election.cc:304] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:47.422482 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ade0f7de597049ffb45274d60c9c0932 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ade0f7de597049ffb45274d60c9c0932 already in progress: creating tablet
I20251026 02:56:47.422799 4098 raft_consensus.cc:2749] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:47.421525 2372 tablet_replica.cc:333] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:47.424705 2372 raft_consensus.cc:2243] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:47.424845 2372 raft_consensus.cc:2272] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:47.425712 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b7162d7bfb2b4b16b958aa22d192d63c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:47.425892 2375 tablet_bootstrap.cc:492] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
W20251026 02:56:47.425884 2195 consensus_peers.cc:597] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dba102838945453cbf44e1cfe505a52c. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:47.426012 2375 ts_tablet_manager.cc:1403] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
W20251026 02:56:47.426240 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7162d7bfb2b4b16b958aa22d192d63c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.426909 2375 raft_consensus.cc:359] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.427090 2375 raft_consensus.cc:385] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.427143 2375 raft_consensus.cc:740] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.427412 2375 consensus_queue.cc:260] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.427945 2375 ts_tablet_manager.cc:1434] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251026 02:56:47.428304 2375 tablet_bootstrap.cc:492] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:47.429841 2375 tablet_bootstrap.cc:654] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.431618 2372 ts_tablet_manager.cc:1916] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:47.440012 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6bf9cedc1f4c46479c8d1b407d1fdc5d" } modify_external_catalogs: true
I20251026 02:56:47.440310 2377 ts_tablet_manager.cc:1916] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:47.440311 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6bf9cedc1f4c46479c8d1b407d1fdc5d" } modify_external_catalogs: true
I20251026 02:56:47.442152 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 99d318aa0d1847c58167bf0af9e75aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:47.449875 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3cf2d84ee15542b888e863fa40b143fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:47.450496 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cf2d84ee15542b888e863fa40b143fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.454223 2378 tablet_bootstrap.cc:654] T 386e1bd4635b4a7e8e7f249b577fd978 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.454782 2381 tablet_bootstrap.cc:492] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:47.454998 2381 ts_tablet_manager.cc:1403] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.001s
I20251026 02:56:47.455855 2381 raft_consensus.cc:359] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.456168 2381 raft_consensus.cc:385] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.456317 2381 raft_consensus.cc:740] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.456669 2381 consensus_queue.cc:260] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.457350 2381 ts_tablet_manager.cc:1434] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:47.457475 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b2bfd9b55c484ef8a9c6345d56f794fb
I20251026 02:56:47.458482 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet b2bfd9b55c484ef8a9c6345d56f794fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:47.458647 2381 tablet_bootstrap.cc:492] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:47.459049 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet b2bfd9b55c484ef8a9c6345d56f794fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:47.460403 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet b2bfd9b55c484ef8a9c6345d56f794fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:47.460752 2381 tablet_bootstrap.cc:654] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.463245 1663 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (table default.Table_30611e161c274e8b99bd1251e5e527aa [id=b5d91d3e8a2643c09fee85976972391a]). Expected version 1 got 0
I20251026 02:56:47.469415 2372 ts_tablet_manager.cc:1929] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:47.469847 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a345fa5edbd48d2891b8b40b09cb3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:47.470707 2372 log.cc:1199] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/34c6a7c774044411a4733b115080c4c6
I20251026 02:56:47.471040 2372 ts_tablet_manager.cc:1950] T 34c6a7c774044411a4733b115080c4c6 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:47.471604 1663 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (table default.Table_30611e161c274e8b99bd1251e5e527aa [id=b5d91d3e8a2643c09fee85976972391a]). Expected version 1 got 0
I20251026 02:56:47.473227 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 5a00bc04b3654f90af342986751344e6 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:47.478561 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 97aa7f0266854c0a8f17e6bceac0c9b5
I20251026 02:56:47.478997 1658 catalog_manager.cc:6719] ReplaceTablet: tablet 97aa7f0266854c0a8f17e6bceac0c9b5 deleted and replaced by tablet 2188765cd2d54d49a0edab1453e0e52a
I20251026 02:56:47.480515 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 97aa7f0266854c0a8f17e6bceac0c9b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 2188765cd2d54d49a0edab1453e0e52a) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:47.480619 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 97aa7f0266854c0a8f17e6bceac0c9b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 2188765cd2d54d49a0edab1453e0e52a) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:47.480846 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 97aa7f0266854c0a8f17e6bceac0c9b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 2188765cd2d54d49a0edab1453e0e52a) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:47.485821 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 663076e362c44837beb322aa965cde42 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:47.487046 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_658f91f2bd254d1285c47121611a34af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:47.470609 2375 tablet_bootstrap.cc:492] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:47.487546 2375 ts_tablet_manager.cc:1403] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
W20251026 02:56:47.487648 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_658f91f2bd254d1285c47121611a34af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.489044 2376 ts_tablet_manager.cc:1929] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251026 02:56:47.490784 2195 consensus_peers.cc:597] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a012d868a78f4324a92db83c81931b9d. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:47.490502 2375 raft_consensus.cc:359] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.491134 2375 raft_consensus.cc:385] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.491238 2375 raft_consensus.cc:740] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.491474 2375 consensus_queue.cc:260] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.492609 2375 ts_tablet_manager.cc:1434] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251026 02:56:47.492947 4067 consensus_queue.cc:1048] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:47.494876 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet f840050f7e314d74ac6e604354a7d4c9
I20251026 02:56:47.496493 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet f840050f7e314d74ac6e604354a7d4c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:47.496866 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet f840050f7e314d74ac6e604354a7d4c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:47.497424 4067 raft_consensus.cc:493] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.497550 4067 raft_consensus.cc:515] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.497896 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c985fc44cad2462f8e82eadf887551c8" } new_table_name: "default.Table_79ee525e9c034edbb93a0f58159e20ce" modify_external_catalogs: true
I20251026 02:56:47.498133 4067 leader_election.cc:290] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:47.498922 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99d318aa0d1847c58167bf0af9e75aa2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:47.499107 2116 raft_consensus.cc:2468] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:47.499697 2195 leader_election.cc:304] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:47.499981 4067 raft_consensus.cc:2804] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:47.500044 4067 raft_consensus.cc:493] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:47.500094 4067 raft_consensus.cc:3060] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.502776 4067 raft_consensus.cc:515] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.503948 4067 leader_election.cc:290] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:47.504577 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99d318aa0d1847c58167bf0af9e75aa2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:47.504748 2116 raft_consensus.cc:3060] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.505151 2381 tablet_bootstrap.cc:492] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:47.505345 2381 ts_tablet_manager.cc:1403] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.001s
I20251026 02:56:47.506726 2381 raft_consensus.cc:359] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.506956 2381 raft_consensus.cc:385] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.507057 2381 raft_consensus.cc:740] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
W20251026 02:56:47.507485 2195 consensus_peers.cc:597] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86e483c52e514f1da54335479f59d68e. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:47.507965 2381 consensus_queue.cc:260] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.509110 2381 ts_tablet_manager.cc:1434] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:47.509259 2116 raft_consensus.cc:2468] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:47.510246 2195 leader_election.cc:304] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:47.510533 4067 raft_consensus.cc:2804] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:47.510673 4067 raft_consensus.cc:697] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:47.510914 4067 consensus_queue.cc:237] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.497485 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet f840050f7e314d74ac6e604354a7d4c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:47.499421 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99d318aa0d1847c58167bf0af9e75aa2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:47.503746 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99d318aa0d1847c58167bf0af9e75aa2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:47.510013 1975 raft_consensus.cc:3060] T 7abcdac4a70b48ad9fbba1bc8a60283b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.511690 2376 log.cc:1199] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/30c619ad37364747b6ebf97a73ef45a2
I20251026 02:56:47.512281 2376 ts_tablet_manager.cc:1950] T 30c619ad37364747b6ebf97a73ef45a2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:47.513666 2381 tablet_bootstrap.cc:492] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:47.513679 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 913) took 4101 ms. Trace:
I20251026 02:56:47.514876 2376 rpcz_store.cc:276] 1026 02:56:43.411905 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.412363 (+ 458us) service_pool.cc:225] Handling call
1026 02:56:47.393767 (+3981404us) ts_tablet_manager.cc:1151] Deleting tablet 30c619ad37364747b6ebf97a73ef45a2
1026 02:56:47.393774 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.449691 (+ 55917us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.488921 (+ 39230us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.513658 (+ 24737us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3981177}
I20251026 02:56:47.516096 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 914) took 3969 ms. Trace:
W20251026 02:56:47.516546 2194 leader_election.cc:343] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.448149 2377 ts_tablet_manager.cc:1929] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251026 02:56:47.516705 2377 log.cc:1199] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3dbe8002df6c4b64b8ec8260ca6b9cf5
I20251026 02:56:47.516938 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 30c619ad37364747b6ebf97a73ef45a2 (table default.Table_ad4b4db01c7d4fe0a4e99f2f54cee0e8 [id=65a13a0482834ea6b52b71c953d61e5d]) successfully deleted
I20251026 02:56:47.517632 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fe09be2527a840b09e3e64c27965845d" } new_table_name: "default.Table_24c7322324534108a75d2fcae7d1082b" modify_external_catalogs: true
I20251026 02:56:47.517799 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a3f3fc3c7a04d05bae28472251f3acc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:47.520092 2375 tablet_bootstrap.cc:492] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:47.520238 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet 086b40036f0c4c97bb95647f91ce7939 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:47.521821 2375 tablet_bootstrap.cc:654] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.522914 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 923) took 5237 ms. Trace:
I20251026 02:56:47.523006 2372 rpcz_store.cc:276] 1026 02:56:42.285450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.285601 (+ 151us) service_pool.cc:225] Handling call
1026 02:56:47.421467 (+5135866us) ts_tablet_manager.cc:1151] Deleting tablet 34c6a7c774044411a4733b115080c4c6
1026 02:56:47.421476 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.435777 (+ 14301us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.469306 (+ 33529us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.522895 (+ 53589us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5118466}
I20251026 02:56:47.523413 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 925) took 4969 ms. Trace:
I20251026 02:56:47.523490 2372 rpcz_store.cc:276] 1026 02:56:42.553515 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.553569 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:47.523279 (+4969710us) ts_tablet_manager.cc:1151] Deleting tablet 5b8e1d79d2a247dba25540daf6b81bf0
1026 02:56:47.523288 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.523399 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4969500}
I20251026 02:56:47.523783 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 926) took 4966 ms. Trace:
I20251026 02:56:47.523855 2372 rpcz_store.cc:276] 1026 02:56:42.557226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.561383 (+ 4157us) service_pool.cc:225] Handling call
1026 02:56:47.523699 (+4962316us) ts_tablet_manager.cc:1151] Deleting tablet 2f197526e7d34f849dbb4f28d8d414cf
1026 02:56:47.523707 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.523772 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4961453}
I20251026 02:56:47.524008 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 34c6a7c774044411a4733b115080c4c6 (table default.Table_6b9199ff4e884b4891446da6e6e6195d [id=d6f9d1f56c154ea3a78a6677347c4a05]) successfully deleted
W20251026 02:56:47.524245 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 5b8e1d79d2a247dba25540daf6b81bf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b8e1d79d2a247dba25540daf6b81bf0 already in progress: creating tablet
W20251026 02:56:47.524466 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 2f197526e7d34f849dbb4f28d8d414cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f197526e7d34f849dbb4f28d8d414cf already in progress: creating tablet
I20251026 02:56:47.526130 2381 tablet_bootstrap.cc:654] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.526257 2377 ts_tablet_manager.cc:1950] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:47.529068 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 912) took 5788 ms. Trace:
I20251026 02:56:47.529237 2377 rpcz_store.cc:276] 1026 02:56:41.740199 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.740737 (+ 538us) service_pool.cc:225] Handling call
1026 02:56:47.419300 (+5678563us) ts_tablet_manager.cc:1151] Deleting tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5
1026 02:56:47.419310 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.444292 (+ 24982us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.448011 (+ 3719us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.529048 (+ 81037us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5678426}
I20251026 02:56:47.530004 2377 tablet_replica.cc:333] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:47.529867 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9b4d5b6ca1484e499853db57b4a4c416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:47.524121 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 928) took 4918 ms. Trace:
W20251026 02:56:47.530403 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b4d5b6ca1484e499853db57b4a4c416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.530725 2377 raft_consensus.cc:2243] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:47.530861 2372 rpcz_store.cc:276] 1026 02:56:42.605317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.605378 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:47.524050 (+4918672us) ts_tablet_manager.cc:1151] Deleting tablet 663076e362c44837beb322aa965cde42
1026 02:56:47.524057 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.524110 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4918476}
I20251026 02:56:47.531453 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 (table default.Table_2dfe8bcc7f1e4933a4174bdab19cffd5 [id=65d576adb71949cc97666624935adbf3]) successfully deleted
I20251026 02:56:47.531693 2377 raft_consensus.cc:2272] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:47.532289 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 929) took 4925 ms. Trace:
W20251026 02:56:47.532276 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 663076e362c44837beb322aa965cde42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 663076e362c44837beb322aa965cde42 already in progress: creating tablet
I20251026 02:56:47.533322 2372 rpcz_store.cc:276] 1026 02:56:42.606636 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.606687 (+ 51us) service_pool.cc:225] Handling call
1026 02:56:47.532133 (+4925446us) ts_tablet_manager.cc:1151] Deleting tablet 8977f824ff374d05b4ed9d88c67329f9
1026 02:56:47.532143 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.532272 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4925236}
I20251026 02:56:47.533651 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 930) took 4922 ms. Trace:
I20251026 02:56:47.533708 2372 rpcz_store.cc:276] 1026 02:56:42.611542 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.611598 (+ 56us) service_pool.cc:225] Handling call
1026 02:56:47.533549 (+4921951us) ts_tablet_manager.cc:1151] Deleting tablet a4f4d7f87916425999574fd5b9f2fdf7
1026 02:56:47.533557 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.533640 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4921707}
I20251026 02:56:47.533926 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 931) took 4919 ms. Trace:
I20251026 02:56:47.533975 2372 rpcz_store.cc:276] 1026 02:56:42.614898 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.614956 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:47.533855 (+4918899us) ts_tablet_manager.cc:1151] Deleting tablet ac82c14e7dcb448b845d998249e2b339
1026 02:56:47.533861 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.533914 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4918674}
I20251026 02:56:47.534183 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 932) took 4908 ms. Trace:
I20251026 02:56:47.534241 2372 rpcz_store.cc:276] 1026 02:56:42.625607 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.625654 (+ 47us) service_pool.cc:225] Handling call
1026 02:56:47.534115 (+4908461us) ts_tablet_manager.cc:1151] Deleting tablet b8fe7825fa8d46669772cf84bc4b3afc
1026 02:56:47.534122 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.534172 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4908250}
I20251026 02:56:47.534442 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 951) took 4493 ms. Trace:
I20251026 02:56:47.534498 2372 rpcz_store.cc:276] 1026 02:56:43.040667 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.040719 (+ 52us) service_pool.cc:225] Handling call
1026 02:56:47.534386 (+4493667us) ts_tablet_manager.cc:1151] Deleting tablet 3b7d85a2941545ceaa16b22e8328b4ff
1026 02:56:47.534393 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.534432 (+ 39us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4493486}
W20251026 02:56:47.534669 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 8977f824ff374d05b4ed9d88c67329f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8977f824ff374d05b4ed9d88c67329f9 already in progress: creating tablet
I20251026 02:56:47.534722 2372 tablet_replica.cc:333] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:47.534888 2372 raft_consensus.cc:2243] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:47.534931 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet ac82c14e7dcb448b845d998249e2b339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac82c14e7dcb448b845d998249e2b339 already in progress: creating tablet
I20251026 02:56:47.535022 2372 raft_consensus.cc:2272] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:47.535089 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet b8fe7825fa8d46669772cf84bc4b3afc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8fe7825fa8d46669772cf84bc4b3afc already in progress: creating tablet
W20251026 02:56:47.535300 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3b7d85a2941545ceaa16b22e8328b4ff because the tablet was not found. No further retry: Not found: Tablet not found: 3b7d85a2941545ceaa16b22e8328b4ff
I20251026 02:56:47.535531 2372 ts_tablet_manager.cc:1916] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:47.536177 2376 rpcz_store.cc:276] 1026 02:56:43.546578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.546627 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:47.515979 (+3969352us) ts_tablet_manager.cc:1151] Deleting tablet bcd5621aa3754733b22508be89f080db
1026 02:56:47.515986 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.516081 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3969169}
I20251026 02:56:47.536780 2377 ts_tablet_manager.cc:1916] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:47.538085 2381 tablet_bootstrap.cc:492] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:47.540890 2381 ts_tablet_manager.cc:1403] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20251026 02:56:47.541918 2378 tablet_bootstrap.cc:492] T 386e1bd4635b4a7e8e7f249b577fd978 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
W20251026 02:56:47.542291 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet bcd5621aa3754733b22508be89f080db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcd5621aa3754733b22508be89f080db already in progress: creating tablet
I20251026 02:56:47.542320 2378 ts_tablet_manager.cc:1403] T 386e1bd4635b4a7e8e7f249b577fd978 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.144s user 0.004s sys 0.008s
I20251026 02:56:47.543097 2378 raft_consensus.cc:359] T 386e1bd4635b4a7e8e7f249b577fd978 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.543308 2378 raft_consensus.cc:385] T 386e1bd4635b4a7e8e7f249b577fd978 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.543256 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_628ae32e1bb94616953205cb1a976e99" } new_table_name: "default.Table_7b3c985f937e4046ba2bf35f5fa2bb44" modify_external_catalogs: true
I20251026 02:56:47.543366 2378 raft_consensus.cc:740] T 386e1bd4635b4a7e8e7f249b577fd978 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.543566 2375 tablet_bootstrap.cc:492] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:47.543555 2378 consensus_queue.cc:260] T 386e1bd4635b4a7e8e7f249b577fd978 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.543679 2375 ts_tablet_manager.cc:1403] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20251026 02:56:47.544046 2378 ts_tablet_manager.cc:1434] T 386e1bd4635b4a7e8e7f249b577fd978 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 02:56:47.544337 2378 tablet_bootstrap.cc:492] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:47.544459 2375 raft_consensus.cc:359] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.544620 2375 raft_consensus.cc:385] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.544682 2375 raft_consensus.cc:740] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.542084 2381 raft_consensus.cc:359] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.545606 2381 raft_consensus.cc:385] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.545694 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 915) took 3875 ms. Trace:
I20251026 02:56:47.546813 2376 rpcz_store.cc:276] 1026 02:56:43.669860 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.670400 (+ 540us) service_pool.cc:225] Handling call
1026 02:56:47.545537 (+3875137us) ts_tablet_manager.cc:1151] Deleting tablet 8fdadd053cc34aeca0d491eb3200207f
1026 02:56:47.545553 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.545672 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3874847}
I20251026 02:56:47.547019 4098 raft_consensus.cc:493] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.547147 4098 raft_consensus.cc:515] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:47.547399 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 8fdadd053cc34aeca0d491eb3200207f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fdadd053cc34aeca0d491eb3200207f already in progress: creating tablet
I20251026 02:56:47.546756 2378 tablet_bootstrap.cc:654] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.547746 4098 leader_election.cc:290] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.548559 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d97084b5914350afbfda5c2a34bf9a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:47.548897 2381 raft_consensus.cc:740] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
W20251026 02:56:47.549347 2053 leader_election.cc:343] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.549837 2381 consensus_queue.cc:260] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.550625 2381 ts_tablet_manager.cc:1434] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251026 02:56:47.551426 2381 tablet_bootstrap.cc:492] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:47.551699 2375 consensus_queue.cc:260] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.555511 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b32c2e53d837476d9a0d5ae8ea1843d0" } new_table_name: "default.Table_bc60cafa54fd4cceb4f7efece8d873b5" modify_external_catalogs: true
I20251026 02:56:47.555867 4009 raft_consensus.cc:493] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.556087 4009 raft_consensus.cc:515] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.556705 4009 leader_election.cc:290] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:47.557192 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d97084b5914350afbfda5c2a34bf9a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:47.557796 2053 leader_election.cc:343] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.558177 2053 leader_election.cc:304] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:47.558594 4098 raft_consensus.cc:2749] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:47.560324 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0bacd15b41149b5856d20068711b432" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:47.560848 2194 leader_election.cc:343] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.562059 2381 tablet_bootstrap.cc:654] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.559764 2375 ts_tablet_manager.cc:1434] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251026 02:56:47.562505 2378 tablet_bootstrap.cc:492] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:47.562533 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0bacd15b41149b5856d20068711b432" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:47.562635 2378 ts_tablet_manager.cc:1403] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20251026 02:56:47.562872 2375 tablet_bootstrap.cc:492] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:47.562996 2116 raft_consensus.cc:2468] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:47.563338 2378 raft_consensus.cc:359] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.563521 2378 raft_consensus.cc:385] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.563577 2378 raft_consensus.cc:740] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.563803 2378 consensus_queue.cc:260] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.564239 2378 ts_tablet_manager.cc:1434] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:47.565022 2378 tablet_bootstrap.cc:492] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:47.566084 2376 tablet_replica.cc:333] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:47.566440 2376 raft_consensus.cc:2243] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 LEADER]: Raft consensus shutting down.
I20251026 02:56:47.566742 2376 pending_rounds.cc:70] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8: Trying to abort 1 pending ops.
I20251026 02:56:47.566689 1658 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (table default.Table_30611e161c274e8b99bd1251e5e527aa [id=b5d91d3e8a2643c09fee85976972391a]). Expected version 1 got 0
I20251026 02:56:47.566946 2376 pending_rounds.cc:77] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7214888579317858304 op_type: NO_OP noop_request { }
I20251026 02:56:47.567138 2376 raft_consensus.cc:2889] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:47.567293 2376 raft_consensus.cc:2272] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:47.568616 2378 tablet_bootstrap.cc:654] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.568560 2376 ts_tablet_manager.cc:1916] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:47.563473 2195 leader_election.cc:304] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:47.571018 4009 raft_consensus.cc:2804] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:47.571122 4009 raft_consensus.cc:493] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:47.571185 4009 raft_consensus.cc:3060] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.572127 2375 tablet_bootstrap.cc:654] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:47.574055 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1104345f80d6418fa907a8a26924c281 consensus state has no leader
I20251026 02:56:47.575259 4009 raft_consensus.cc:515] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.576450 4009 leader_election.cc:290] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:47.578236 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_bdf9ad8ed992475dab446da6038be0b3" } modify_external_catalogs: true
I20251026 02:56:47.578323 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0bacd15b41149b5856d20068711b432" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:47.578413 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_bdf9ad8ed992475dab446da6038be0b3" } modify_external_catalogs: true
I20251026 02:56:47.578465 2116 raft_consensus.cc:3060] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.580508 4009 consensus_queue.cc:1048] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:47.584017 2378 tablet_bootstrap.cc:492] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:47.584184 2378 ts_tablet_manager.cc:1403] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251026 02:56:47.584684 4098 raft_consensus.cc:493] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.584808 4098 raft_consensus.cc:515] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.584950 2378 raft_consensus.cc:359] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.585085 2378 raft_consensus.cc:385] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.585131 2378 raft_consensus.cc:740] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.585413 4098 leader_election.cc:290] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.585361 2378 consensus_queue.cc:260] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.585852 2378 ts_tablet_manager.cc:1434] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:47.586635 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1d8bfca6a554e2abbbb9a0c7e2c0c94" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:47.587246 2053 leader_election.cc:343] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.588158 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1d8bfca6a554e2abbbb9a0c7e2c0c94" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:47.588663 2053 leader_election.cc:343] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.588768 2053 leader_election.cc:304] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:47.589170 4098 raft_consensus.cc:2749] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:47.590855 2381 tablet_bootstrap.cc:492] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:47.592531 2116 raft_consensus.cc:2468] T c0bacd15b41149b5856d20068711b432 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:47.591955 2381 ts_tablet_manager.cc:1403] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20251026 02:56:47.592924 2195 leader_election.cc:304] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:47.593278 4068 raft_consensus.cc:2804] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:47.593384 4068 raft_consensus.cc:697] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:47.593608 4068 consensus_queue.cc:237] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.596556 2378 tablet_bootstrap.cc:492] T 4db275efdd344817b11a2168675cf95e P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:47.598942 2378 tablet_bootstrap.cc:654] T 4db275efdd344817b11a2168675cf95e P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.599886 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0bacd15b41149b5856d20068711b432" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:47.600502 2194 leader_election.cc:343] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.611747 2377 ts_tablet_manager.cc:1929] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:47.612674 2375 tablet_bootstrap.cc:492] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:47.621704 2375 ts_tablet_manager.cc:1403] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.001s
I20251026 02:56:47.622779 2375 raft_consensus.cc:359] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.623342 2377 log.cc:1199] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/7177931353164b2aa82c00b2807ccd63
I20251026 02:56:47.623903 2377 ts_tablet_manager.cc:1950] T 7177931353164b2aa82c00b2807ccd63 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:47.626319 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 911) took 5886 ms. Trace:
I20251026 02:56:47.626253 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7c080afce3474539b73dd8c734adef21" } modify_external_catalogs: true
I20251026 02:56:47.626468 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7c080afce3474539b73dd8c734adef21" } modify_external_catalogs: true
I20251026 02:56:47.626609 2375 raft_consensus.cc:385] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.626977 2375 raft_consensus.cc:740] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.627318 2375 consensus_queue.cc:260] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.628147 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_acb7127a0c1946b79991b8e0f61352bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:47.628867 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acb7127a0c1946b79991b8e0f61352bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.629280 2377 rpcz_store.cc:276] 1026 02:56:41.740087 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.741127 (+ 1040us) service_pool.cc:225] Handling call
1026 02:56:47.529958 (+5788831us) ts_tablet_manager.cc:1151] Deleting tablet 7177931353164b2aa82c00b2807ccd63
1026 02:56:47.529968 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.553970 (+ 24002us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.611619 (+ 57649us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.626296 (+ 14677us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5788673}
I20251026 02:56:47.615100 2372 ts_tablet_manager.cc:1929] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:47.634245 2372 log.cc:1199] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/99d318aa0d1847c58167bf0af9e75aa2
I20251026 02:56:47.634846 1975 raft_consensus.cc:3060] T 386e1bd4635b4a7e8e7f249b577fd978 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.621040 2381 raft_consensus.cc:359] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.636093 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 7177931353164b2aa82c00b2807ccd63 (table default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 [id=7de20c0869584b56b936144d62202444]) successfully deleted
I20251026 02:56:47.636180 2381 raft_consensus.cc:385] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.636785 2381 raft_consensus.cc:740] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.637100 2381 consensus_queue.cc:260] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.637795 2381 ts_tablet_manager.cc:1434] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20251026 02:56:47.638149 2381 tablet_bootstrap.cc:492] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:47.638531 2378 tablet_bootstrap.cc:492] T 4db275efdd344817b11a2168675cf95e P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:47.638681 2378 ts_tablet_manager.cc:1403] T 4db275efdd344817b11a2168675cf95e P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.002s
I20251026 02:56:47.639243 4009 consensus_queue.cc:1048] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:47.639504 2378 raft_consensus.cc:359] T 4db275efdd344817b11a2168675cf95e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.639660 2378 raft_consensus.cc:385] T 4db275efdd344817b11a2168675cf95e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.639714 2378 raft_consensus.cc:740] T 4db275efdd344817b11a2168675cf95e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.639905 2378 consensus_queue.cc:260] T 4db275efdd344817b11a2168675cf95e P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.640369 2378 ts_tablet_manager.cc:1434] T 4db275efdd344817b11a2168675cf95e P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:47.640630 2378 tablet_bootstrap.cc:492] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:47.646844 2378 tablet_bootstrap.cc:654] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.655516 1982 raft_consensus.cc:3060] T 48105e54d6324622a032e1288a46188c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:47.657929 1648 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 13acb028b89e47eeadff91fe61cbebf3 (table default.Table_791ef632f7b24cf2a51f8de05d259ad5 [id=f6b9a96f766f491fa6967954c38aaa2b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 13acb028b89e47eeadff91fe61cbebf3 consensus state has no leader
I20251026 02:56:47.634090 2375 ts_tablet_manager.cc:1434] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251026 02:56:47.659998 2375 tablet_bootstrap.cc:492] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:47.660243 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 908) took 5920 ms. Trace:
I20251026 02:56:47.661351 2377 rpcz_store.cc:276] 1026 02:56:41.739868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.749889 (+ 10021us) service_pool.cc:225] Handling call
1026 02:56:47.659315 (+5909426us) ts_tablet_manager.cc:1151] Deleting tablet ed836bb7519e4b0290e6547fa57128bc
1026 02:56:47.659327 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.660190 (+ 863us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5909279}
I20251026 02:56:47.664536 2375 tablet_bootstrap.cc:654] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:47.662714 2195 consensus_peers.cc:597] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34c6a7c774044411a4733b115080c4c6. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:47.662796 4068 consensus_queue.cc:1048] T 48105e54d6324622a032e1288a46188c P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:47.668690 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e427e0ebe73472b83af7a9cae6157d2 has no consensus state
I20251026 02:56:47.669088 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 907) took 5929 ms. Trace:
I20251026 02:56:47.669173 2377 rpcz_store.cc:276] 1026 02:56:41.739788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.751350 (+ 11562us) service_pool.cc:225] Handling call
1026 02:56:47.668924 (+5917574us) ts_tablet_manager.cc:1151] Deleting tablet 887dc91626664f23b39cf70a6bb9381e
1026 02:56:47.668935 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.669065 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5917394}
W20251026 02:56:47.669917 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ed836bb7519e4b0290e6547fa57128bc because the tablet was not found. No further retry: Not found: Tablet not found: ed836bb7519e4b0290e6547fa57128bc
I20251026 02:56:47.670083 2378 tablet_bootstrap.cc:492] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:47.670219 2377 tablet_replica.cc:333] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:47.670212 2378 ts_tablet_manager.cc:1403] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
W20251026 02:56:47.670183 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 887dc91626664f23b39cf70a6bb9381e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 887dc91626664f23b39cf70a6bb9381e already in progress: creating tablet
I20251026 02:56:47.670436 2377 raft_consensus.cc:2243] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:47.670984 2378 raft_consensus.cc:359] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.671136 2378 raft_consensus.cc:385] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.671178 2378 raft_consensus.cc:740] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.671386 2378 consensus_queue.cc:260] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.671880 2378 ts_tablet_manager.cc:1434] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:47.671581 2381 tablet_bootstrap.cc:654] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.675320 2377 raft_consensus.cc:2272] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:47.676762 2378 tablet_bootstrap.cc:492] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:47.661177 2372 ts_tablet_manager.cc:1950] T 99d318aa0d1847c58167bf0af9e75aa2 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:47.677778 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 949) took 4637 ms. Trace:
I20251026 02:56:47.677984 2372 rpcz_store.cc:276] 1026 02:56:43.040421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.040834 (+ 413us) service_pool.cc:225] Handling call
1026 02:56:47.534686 (+4493852us) ts_tablet_manager.cc:1151] Deleting tablet 99d318aa0d1847c58167bf0af9e75aa2
1026 02:56:47.534694 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.572996 (+ 38302us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.614988 (+ 41992us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.677707 (+ 62719us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4493733}
I20251026 02:56:47.678445 2378 tablet_bootstrap.cc:654] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.678589 2372 tablet_replica.cc:333] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:47.678637 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 99d318aa0d1847c58167bf0af9e75aa2 (table default.Table_62fd5a902d0648cea484f259d02de5f9 [id=3dc4d9941b7147adb84d1754e2783aa9]) successfully deleted
I20251026 02:56:47.678853 2372 raft_consensus.cc:2243] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:47.679050 2372 raft_consensus.cc:2272] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:47.685542 1982 raft_consensus.cc:3060] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.686342 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6b7a7d49a4cf4c7084d664fa624c3da4" } new_table_name: "default.Table_d535c2384403424891ff8d3d84045371" modify_external_catalogs: true
I20251026 02:56:47.690991 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6b4fcf2a2bad4ca7ae3d9bce7d17c222"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:47.691731 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b4fcf2a2bad4ca7ae3d9bce7d17c222 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.701601 1658 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (table default.Table_30611e161c274e8b99bd1251e5e527aa [id=b5d91d3e8a2643c09fee85976972391a]). Expected version 1 got 0
I20251026 02:56:47.701864 1658 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 23062e7292f44ad5ad9dfc22a401741c (table default.Table_2cf16843709c4e2a9c3632c25c6bbe25 [id=2e17bae10e0b48a492f5ee0bed7d00d0]). Expected version 2 got 0
I20251026 02:56:47.701988 1658 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 386e1bd4635b4a7e8e7f249b577fd978 (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]). Expected version 2 got 0
I20251026 02:56:47.703083 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c29fa2ff53174902a60e57cb90cb11c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:47.703240 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 4db275efdd344817b11a2168675cf95e (table default.Table_c30cfa52b6524323bda478613d74178b [id=1fd74186ea10498599435dccd7330d25]). Expected version 1 got 0
I20251026 02:56:47.703426 1660 catalog_manager.cc:5649] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 2, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 2 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:47.703764 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 2fd8f02126b24285bbfa575540ba0f09 (table default.Table_7539d36a088f4eafaa21d151ed9ea1cb [id=1950d7cb826b4a99b1ba8866e1d845f3]). Expected version 1 got 0
I20251026 02:56:47.706765 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 41ec602163d04a7b8720a262e8f85a56 (DEFAULT_TABLE table=default.Table_191c164a60554219a63000aad6c771b4 [id=966d706e56b949d2adc72ab7bb05df51]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.707185 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ec602163d04a7b8720a262e8f85a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.705806 2096 tablet_service.cc:1505] Processing CreateTablet for tablet a66b9ee835c5406b87799317737a844a (DEFAULT_TABLE table=default.Table_9f9c766d51344e4683974de3d9d367e0 [id=116c717643c348458c17564b3062d6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.707746 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66b9ee835c5406b87799317737a844a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.711715 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ce1ab21ddf8e45bc9063b63db9c1b97f
I20251026 02:56:47.712601 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 8ce64983451941fc82d74a6da34b11ed (DEFAULT_TABLE table=default.Table_60141a907ea64450acd7914381c35889 [id=a6daff2379934c68a817db767b2df67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.713086 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce64983451941fc82d74a6da34b11ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.716405 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ab21ddf8e45bc9063b63db9c1b97f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:47.718703 1978 raft_consensus.cc:3060] T 4db275efdd344817b11a2168675cf95e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.719182 1960 tablet_service.cc:1505] Processing CreateTablet for tablet a66b9ee835c5406b87799317737a844a (DEFAULT_TABLE table=default.Table_9f9c766d51344e4683974de3d9d367e0 [id=116c717643c348458c17564b3062d6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.719733 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 41ec602163d04a7b8720a262e8f85a56 (DEFAULT_TABLE table=default.Table_191c164a60554219a63000aad6c771b4 [id=966d706e56b949d2adc72ab7bb05df51]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:47.720283 1656 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e044af9739364ecdbbaddc115c09bfab consensus state has no leader
I20251026 02:56:47.720499 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 8ce64983451941fc82d74a6da34b11ed (DEFAULT_TABLE table=default.Table_60141a907ea64450acd7914381c35889 [id=a6daff2379934c68a817db767b2df67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.720988 1951 tablet_service.cc:1505] Processing CreateTablet for tablet d41b0de113324989b1b9e85f337b550c (DEFAULT_TABLE table=default.Table_234823813b084151a8c49ab91bf0d8f7 [id=99526a43f68c4acebd45d5e367647d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.721341 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ab21ddf8e45bc9063b63db9c1b97f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:47.703599 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c29fa2ff53174902a60e57cb90cb11c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:47.725275 4150 tablet.cc:1722] T e9cd10092aac40068a1c7f85e514fd37 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:47.731592 1955 tablet_service.cc:1505] Processing CreateTablet for tablet c24fdefde360460487a92b9bc90ca37c (DEFAULT_TABLE table=default.Table_810dcd19177646f483aeef93df04b9ec [id=6a009e650eff4c739702d76fad6880c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.732002 2082 tablet_service.cc:1505] Processing CreateTablet for tablet c24fdefde360460487a92b9bc90ca37c (DEFAULT_TABLE table=default.Table_810dcd19177646f483aeef93df04b9ec [id=6a009e650eff4c739702d76fad6880c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.732457 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24fdefde360460487a92b9bc90ca37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.732497 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_082dd939577943e5ae1d59dea699d4ca" } modify_external_catalogs: true
I20251026 02:56:47.732715 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_082dd939577943e5ae1d59dea699d4ca" } modify_external_catalogs: true
I20251026 02:56:47.734892 2242 tablet_service.cc:1505] Processing CreateTablet for tablet a66b9ee835c5406b87799317737a844a (DEFAULT_TABLE table=default.Table_9f9c766d51344e4683974de3d9d367e0 [id=116c717643c348458c17564b3062d6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.735271 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66b9ee835c5406b87799317737a844a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.735647 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ac6c5e4166e14e079228d171f79e964c" } modify_external_catalogs: true
I20251026 02:56:47.735818 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ac6c5e4166e14e079228d171f79e964c" } modify_external_catalogs: true
I20251026 02:56:47.738327 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66b9ee835c5406b87799317737a844a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.739609 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24fdefde360460487a92b9bc90ca37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.740341 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d41b0de113324989b1b9e85f337b550c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.674603 2376 ts_tablet_manager.cc:1929] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251026 02:56:47.676082 4089 raft_consensus.cc:493] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.742439 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 4e427e0ebe73472b83af7a9cae6157d2 (DEFAULT_TABLE table=default.Table_7b3c985f937e4046ba2bf35f5fa2bb44 [id=e579b19665394b56816a555268462716]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.742869 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e427e0ebe73472b83af7a9cae6157d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.743434 4089 raft_consensus.cc:515] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.743682 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ec602163d04a7b8720a262e8f85a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.744114 4089 leader_election.cc:290] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:47.745318 1916 leader_election.cc:343] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: ed836bb7519e4b0290e6547fa57128bc
I20251026 02:56:47.749176 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 4e427e0ebe73472b83af7a9cae6157d2 (DEFAULT_TABLE table=default.Table_7b3c985f937e4046ba2bf35f5fa2bb44 [id=e579b19665394b56816a555268462716]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.749959 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce64983451941fc82d74a6da34b11ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.749922 2098 tablet_service.cc:1505] Processing CreateTablet for tablet d41b0de113324989b1b9e85f337b550c (DEFAULT_TABLE table=default.Table_234823813b084151a8c49ab91bf0d8f7 [id=99526a43f68c4acebd45d5e367647d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.750293 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d41b0de113324989b1b9e85f337b550c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.754176 2376 log.cc:1199] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/b1ca5b8379c94deab1130bfe8bdc87b0
I20251026 02:56:47.755899 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e427e0ebe73472b83af7a9cae6157d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.757493 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 396c126f5f7d4505bf0b7ece5daf7a79 (DEFAULT_TABLE table=default.Table_f83eb680ec4a4220a61c02921eac359a [id=b3a75783a642420abcce39eaaf2981f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.757869 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 396c126f5f7d4505bf0b7ece5daf7a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.758155 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 396c126f5f7d4505bf0b7ece5daf7a79 (DEFAULT_TABLE table=default.Table_f83eb680ec4a4220a61c02921eac359a [id=b3a75783a642420abcce39eaaf2981f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.758540 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 396c126f5f7d4505bf0b7ece5daf7a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.762295 2377 ts_tablet_manager.cc:1916] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:47.762961 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 2efe2fc1234e4e309ab3ef94e4c3217a (DEFAULT_TABLE table=default.Table_141d324fb96a4df0b7916345b94ac27d [id=a1db083d2f1541929aef0fc57805999e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.763386 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efe2fc1234e4e309ab3ef94e4c3217a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.763859 2376 ts_tablet_manager.cc:1950] T b1ca5b8379c94deab1130bfe8bdc87b0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:47.762926 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 41ec602163d04a7b8720a262e8f85a56 (DEFAULT_TABLE table=default.Table_191c164a60554219a63000aad6c771b4 [id=966d706e56b949d2adc72ab7bb05df51]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.764251 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 8ce64983451941fc82d74a6da34b11ed (DEFAULT_TABLE table=default.Table_60141a907ea64450acd7914381c35889 [id=a6daff2379934c68a817db767b2df67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.764757 2227 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ab21ddf8e45bc9063b63db9c1b97f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:47.764780 2231 tablet_service.cc:1505] Processing CreateTablet for tablet d41b0de113324989b1b9e85f337b550c (DEFAULT_TABLE table=default.Table_234823813b084151a8c49ab91bf0d8f7 [id=99526a43f68c4acebd45d5e367647d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.765174 2226 tablet_service.cc:1505] Processing CreateTablet for tablet c24fdefde360460487a92b9bc90ca37c (DEFAULT_TABLE table=default.Table_810dcd19177646f483aeef93df04b9ec [id=6a009e650eff4c739702d76fad6880c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.765627 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 4e427e0ebe73472b83af7a9cae6157d2 (DEFAULT_TABLE table=default.Table_7b3c985f937e4046ba2bf35f5fa2bb44 [id=e579b19665394b56816a555268462716]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.765818 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 396c126f5f7d4505bf0b7ece5daf7a79 (DEFAULT_TABLE table=default.Table_f83eb680ec4a4220a61c02921eac359a [id=b3a75783a642420abcce39eaaf2981f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.765880 2238 tablet_service.cc:1505] Processing CreateTablet for tablet 2efe2fc1234e4e309ab3ef94e4c3217a (DEFAULT_TABLE table=default.Table_141d324fb96a4df0b7916345b94ac27d [id=a1db083d2f1541929aef0fc57805999e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.767069 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ec602163d04a7b8720a262e8f85a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.767313 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e427e0ebe73472b83af7a9cae6157d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.772549 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 396c126f5f7d4505bf0b7ece5daf7a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.772629 4155 tablet.cc:1722] T e9cd10092aac40068a1c7f85e514fd37 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:47.773651 2381 tablet_bootstrap.cc:492] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:47.773851 2381 ts_tablet_manager.cc:1403] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.136s user 0.002s sys 0.002s
I20251026 02:56:47.774780 2381 raft_consensus.cc:359] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.775029 2381 raft_consensus.cc:385] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.775111 2381 raft_consensus.cc:740] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.775383 2381 consensus_queue.cc:260] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.776126 2381 ts_tablet_manager.cc:1434] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:47.776461 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efe2fc1234e4e309ab3ef94e4c3217a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.776573 2372 ts_tablet_manager.cc:1916] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:47.780447 4068 raft_consensus.cc:493] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.780483 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24fdefde360460487a92b9bc90ca37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.780694 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce64983451941fc82d74a6da34b11ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.784560 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d41b0de113324989b1b9e85f337b550c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.784826 4009 raft_consensus.cc:493] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.784945 4009 raft_consensus.cc:515] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.785528 4009 leader_election.cc:290] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:47.788084 4068 raft_consensus.cc:515] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.788074 2092 tablet_service.cc:1505] Processing CreateTablet for tablet a6f15aab31204d75a413192d4e6581de (DEFAULT_TABLE table=default.Table_3ff405b3d7584fa9b587d962606e80c2 [id=77b8d071da684c8abf42467b7e96fbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.788600 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d43b6f22dd4f4f9c9657f21514550312" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:47.788811 2110 raft_consensus.cc:2468] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:47.789026 4068 leader_election.cc:290] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:47.789446 2195 leader_election.cc:304] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:47.789762 4068 raft_consensus.cc:2804] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:47.789829 4068 raft_consensus.cc:493] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:47.789891 4068 raft_consensus.cc:3060] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.790139 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 7847c12059f440cf83e6f1c7dc0f6808 (DEFAULT_TABLE table=default.Table_5b3d81865782468ca8ab3c0023b31352 [id=93272ff7cfda4b4dba4c6935ce6d6534]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.791174 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a3f3fc3c7a04d05bae28472251f3acc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:47.791350 2110 raft_consensus.cc:2468] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:47.791679 2195 leader_election.cc:304] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:47.791980 4069 raft_consensus.cc:2804] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:47.792048 4069 raft_consensus.cc:493] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:47.792101 4069 raft_consensus.cc:3060] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.793673 4068 raft_consensus.cc:515] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.794610 4069 raft_consensus.cc:515] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.795114 4069 leader_election.cc:290] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:47.795481 4068 leader_election.cc:290] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:47.796027 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d43b6f22dd4f4f9c9657f21514550312" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:47.796239 2116 raft_consensus.cc:3060] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:47.766360 1915 leader_election.cc:343] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: ed836bb7519e4b0290e6547fa57128bc
I20251026 02:56:47.780603 4157 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251026 02:56:47.800462 1915 leader_election.cc:304] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:47.800765 4089 raft_consensus.cc:2749] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:47.789141 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a3f3fc3c7a04d05bae28472251f3acc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:47.800982 4089 raft_consensus.cc:493] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.794651 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d43b6f22dd4f4f9c9657f21514550312" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:47.801147 4089 raft_consensus.cc:515] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.801808 4089 leader_election.cc:290] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:47.802433 2194 leader_election.cc:343] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:47.802446 1916 leader_election.cc:343] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 5c6e3e47b3e7421e945cfbe671b06abe
W20251026 02:56:47.802767 2194 leader_election.cc:343] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.794843 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d43b6f22dd4f4f9c9657f21514550312" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:47.795617 1973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a3f3fc3c7a04d05bae28472251f3acc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:47.803417 1915 leader_election.cc:343] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 5c6e3e47b3e7421e945cfbe671b06abe
I20251026 02:56:47.803539 1915 leader_election.cc:304] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
W20251026 02:56:47.803743 2194 leader_election.cc:343] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.803956 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a3f3fc3c7a04d05bae28472251f3acc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
W20251026 02:56:47.804049 2194 leader_election.cc:343] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.804117 2110 raft_consensus.cc:3060] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.804286 4095 raft_consensus.cc:2749] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:47.806145 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 28a4cf51159f4f1c80f03954d2fc34e1 (DEFAULT_TABLE table=default.Table_d0ccee4f250840c49e212c09b25cb0d3 [id=c3387a3ca9c84b09a37ce25a0d63aecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.806843 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 07b32f9198a1497c9d76043aec1c0696 (DEFAULT_TABLE table=default.Table_4e364f533870440b9c9390bb6091228e [id=bb13015c865742e1996fe19592dad49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.816286 1956 tablet_service.cc:1505] Processing CreateTablet for tablet a6f15aab31204d75a413192d4e6581de (DEFAULT_TABLE table=default.Table_3ff405b3d7584fa9b587d962606e80c2 [id=77b8d071da684c8abf42467b7e96fbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.816753 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 07b32f9198a1497c9d76043aec1c0696 (DEFAULT_TABLE table=default.Table_4e364f533870440b9c9390bb6091228e [id=bb13015c865742e1996fe19592dad49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.817058 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 28a4cf51159f4f1c80f03954d2fc34e1 (DEFAULT_TABLE table=default.Table_d0ccee4f250840c49e212c09b25cb0d3 [id=c3387a3ca9c84b09a37ce25a0d63aecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.817353 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 56eda85da4724dfcbc14ef8fe8745fe9 (DEFAULT_TABLE table=default.Table_ef4b978686a546d896741fa10badbddd [id=645302bdeb7748bc9298cf908d294c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.817915 4155 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251026 02:56:47.818598 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 56eda85da4724dfcbc14ef8fe8745fe9 (DEFAULT_TABLE table=default.Table_ef4b978686a546d896741fa10badbddd [id=645302bdeb7748bc9298cf908d294c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.820343 2110 raft_consensus.cc:2468] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:47.821535 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 56eda85da4724dfcbc14ef8fe8745fe9 (DEFAULT_TABLE table=default.Table_ef4b978686a546d896741fa10badbddd [id=645302bdeb7748bc9298cf908d294c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.821979 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56eda85da4724dfcbc14ef8fe8745fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.822638 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 36dcd848d7454d8dae946f05b6e883a1 (DEFAULT_TABLE table=default.Table_73427fdee7c040a0a8dc618e777f1182 [id=e4250f6c594f49818692b819437cd1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.821535 2227 tablet_service.cc:1505] Processing CreateTablet for tablet 28a4cf51159f4f1c80f03954d2fc34e1 (DEFAULT_TABLE table=default.Table_d0ccee4f250840c49e212c09b25cb0d3 [id=c3387a3ca9c84b09a37ce25a0d63aecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.824671 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28a4cf51159f4f1c80f03954d2fc34e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.826674 2084 tablet_service.cc:1505] Processing CreateTablet for tablet d9c641bcc9c945dc934474f42828fb79 (DEFAULT_TABLE table=default.Table_bc05ce655d584c6480d446730692ceeb [id=1b3cd64d9f6b4eeca468545a71e1da73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.788576 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f15aab31204d75a413192d4e6581de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.835843 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9c641bcc9c945dc934474f42828fb79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.836072 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7847c12059f440cf83e6f1c7dc0f6808. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.839782 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36dcd848d7454d8dae946f05b6e883a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.840348 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 07b32f9198a1497c9d76043aec1c0696 (DEFAULT_TABLE table=default.Table_4e364f533870440b9c9390bb6091228e [id=bb13015c865742e1996fe19592dad49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.840718 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07b32f9198a1497c9d76043aec1c0696. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.845193 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 7847c12059f440cf83e6f1c7dc0f6808 (DEFAULT_TABLE table=default.Table_5b3d81865782468ca8ab3c0023b31352 [id=93272ff7cfda4b4dba4c6935ce6d6534]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.849488 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56eda85da4724dfcbc14ef8fe8745fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.850039 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 36dcd848d7454d8dae946f05b6e883a1 (DEFAULT_TABLE table=default.Table_73427fdee7c040a0a8dc618e777f1182 [id=e4250f6c594f49818692b819437cd1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.850173 1945 tablet_service.cc:1505] Processing CreateTablet for tablet d9c641bcc9c945dc934474f42828fb79 (DEFAULT_TABLE table=default.Table_bc05ce655d584c6480d446730692ceeb [id=1b3cd64d9f6b4eeca468545a71e1da73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.851531 2195 leader_election.cc:304] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:47.851842 4009 raft_consensus.cc:2804] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:47.851934 4009 raft_consensus.cc:697] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:47.852198 4009 consensus_queue.cc:237] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.854440 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28a4cf51159f4f1c80f03954d2fc34e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.855043 2226 tablet_service.cc:1505] Processing CreateTablet for tablet 7847c12059f440cf83e6f1c7dc0f6808 (DEFAULT_TABLE table=default.Table_5b3d81865782468ca8ab3c0023b31352 [id=93272ff7cfda4b4dba4c6935ce6d6534]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.855463 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7847c12059f440cf83e6f1c7dc0f6808. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.859752 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07b32f9198a1497c9d76043aec1c0696. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.863794 2238 tablet_service.cc:1505] Processing CreateTablet for tablet a6f15aab31204d75a413192d4e6581de (DEFAULT_TABLE table=default.Table_3ff405b3d7584fa9b587d962606e80c2 [id=77b8d071da684c8abf42467b7e96fbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.864259 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f15aab31204d75a413192d4e6581de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.865576 4095 raft_consensus.cc:493] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.865721 4095 raft_consensus.cc:515] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.866387 4095 leader_election.cc:290] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.867002 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23062e7292f44ad5ad9dfc22a401741c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:47.867548 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23062e7292f44ad5ad9dfc22a401741c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:47.866988 4148 tablet.cc:1722] T e9cd10092aac40068a1c7f85e514fd37 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:47.868000 1915 leader_election.cc:304] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:47.868287 4095 raft_consensus.cc:2749] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:47.869011 3997 raft_consensus.cc:493] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.869195 2375 tablet_bootstrap.cc:492] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:47.869333 2375 ts_tablet_manager.cc:1403] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.209s user 0.003s sys 0.000s
I20251026 02:56:47.869195 3997 raft_consensus.cc:515] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.869977 2375 raft_consensus.cc:359] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.870121 2375 raft_consensus.cc:385] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.870167 2375 raft_consensus.cc:740] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.870375 2375 consensus_queue.cc:260] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.870612 3997 leader_election.cc:290] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.870961 2375 ts_tablet_manager.cc:1434] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:47.871061 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fb42b54f558499090a1d7cf257dd84c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:47.871606 2375 tablet_bootstrap.cc:492] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:47.871573 2053 leader_election.cc:343] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.872498 1981 raft_consensus.cc:3060] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.876500 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fb42b54f558499090a1d7cf257dd84c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:47.876870 4148 tablet.cc:1722] T 5226d4516ce24ecda51dab35c86da132 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251026 02:56:47.877072 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28a4cf51159f4f1c80f03954d2fc34e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:47.877131 2053 leader_election.cc:343] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.877260 2053 leader_election.cc:304] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:47.877648 3997 raft_consensus.cc:2749] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:47.881605 3997 raft_consensus.cc:493] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.882002 3997 raft_consensus.cc:515] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.882925 3997 leader_election.cc:290] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.883140 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e89a98fcfdbb4011876b0ebc2af91f67" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:47.883684 2053 leader_election.cc:343] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.886605 3997 raft_consensus.cc:493] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.886811 3997 raft_consensus.cc:515] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.887426 3997 leader_election.cc:290] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.887876 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfd3e23d83a84648bd11cd258377aecb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:47.888366 2053 leader_election.cc:343] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.890303 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f15aab31204d75a413192d4e6581de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.890502 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56eda85da4724dfcbc14ef8fe8745fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.890900 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9c641bcc9c945dc934474f42828fb79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.895488 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07b32f9198a1497c9d76043aec1c0696. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.895661 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36dcd848d7454d8dae946f05b6e883a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.903493 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 917) took 4207 ms. Trace:
I20251026 02:56:47.903673 2376 rpcz_store.cc:276] 1026 02:56:43.695661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.695720 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:47.547740 (+3852020us) ts_tablet_manager.cc:1151] Deleting tablet b1ca5b8379c94deab1130bfe8bdc87b0
1026 02:56:47.555661 (+ 7921us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.619690 (+ 64029us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.674485 (+ 54795us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.903471 (+228986us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3851801}
I20251026 02:56:47.904270 2376 tablet_replica.cc:333] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:47.904512 2376 raft_consensus.cc:2243] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:47.904865 2376 raft_consensus.cc:2272] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:47.905414 2376 ts_tablet_manager.cc:1916] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:47.908824 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 6667ddda96fb465ebdf54560f04c5555 (DEFAULT_TABLE table=default.Table_dbd53db2b8684c5ba304639f97a47556 [id=98e5d2bda13a42859a7d33fcd0cacfdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.909289 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6667ddda96fb465ebdf54560f04c5555. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.910071 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 6667ddda96fb465ebdf54560f04c5555 (DEFAULT_TABLE table=default.Table_dbd53db2b8684c5ba304639f97a47556 [id=98e5d2bda13a42859a7d33fcd0cacfdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.910480 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6667ddda96fb465ebdf54560f04c5555. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.914469 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 19de884da8b64d8eaebeaae646664286 (DEFAULT_TABLE table=default.Table_2f7abb9f1aaf4a52b3c0066e12817faf [id=35816ac0236a4d6ca2786cca66603bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.914983 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19de884da8b64d8eaebeaae646664286. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.917486 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 73f0b1443e2c4294a222e62433f3b435 (DEFAULT_TABLE table=default.Table_0982fce6dbaa41f69b6e80e254f245db [id=39a1785e03fd45b7a8ea0f14bf5a362a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.917572 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet b1ca5b8379c94deab1130bfe8bdc87b0 (table default.Table_85c523c971b34e68bed4afa5d1aac450 [id=32c06e4782a8441c8d1614e81277b52e]) successfully deleted
I20251026 02:56:47.917919 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f0b1443e2c4294a222e62433f3b435. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.918654 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 58043f2dd48d4fffa613b3693e13dc81 (DEFAULT_TABLE table=default.Table_4c255180d55843908eaf691ea0b0a6ec [id=9c30114a0a1f4b96993cdc043f95fc20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.919049 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58043f2dd48d4fffa613b3693e13dc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.920634 4095 raft_consensus.cc:493] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.920765 4095 raft_consensus.cc:515] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.921384 4095 leader_election.cc:290] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:47.922206 1916 leader_election.cc:343] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 62d30d79a6e4432a8dbe40a104e87fa0
I20251026 02:56:47.923229 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 2efe2fc1234e4e309ab3ef94e4c3217a (DEFAULT_TABLE table=default.Table_141d324fb96a4df0b7916345b94ac27d [id=a1db083d2f1541929aef0fc57805999e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.923616 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efe2fc1234e4e309ab3ef94e4c3217a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.927285 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 58043f2dd48d4fffa613b3693e13dc81 (DEFAULT_TABLE table=default.Table_4c255180d55843908eaf691ea0b0a6ec [id=9c30114a0a1f4b96993cdc043f95fc20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.928969 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58043f2dd48d4fffa613b3693e13dc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.931428 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 19de884da8b64d8eaebeaae646664286 (DEFAULT_TABLE table=default.Table_2f7abb9f1aaf4a52b3c0066e12817faf [id=35816ac0236a4d6ca2786cca66603bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.931819 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19de884da8b64d8eaebeaae646664286. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.932749 4155 tablet.cc:1722] T 4db275efdd344817b11a2168675cf95e P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:47.933190 2100 tablet_service.cc:1505] Processing CreateTablet for tablet fbb5e30694d2411f8c1932297837cf67 (DEFAULT_TABLE table=default.Table_089d38ef94b2431f926063d1eb24d294 [id=76f302ac2e2d4907b1984d19fa1ab1be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.934051 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbb5e30694d2411f8c1932297837cf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.935438 2372 ts_tablet_manager.cc:1929] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:47.935542 2372 log.cc:1199] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/9a345fa5edbd48d2891b8b40b09cb3f8
I20251026 02:56:47.935830 2372 ts_tablet_manager.cc:1950] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:47.940416 2375 tablet_bootstrap.cc:654] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:47.940560 2376 ts_tablet_manager.cc:1929] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:47.883983 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e89a98fcfdbb4011876b0ebc2af91f67" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:47.886322 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 36dcd848d7454d8dae946f05b6e883a1 (DEFAULT_TABLE table=default.Table_73427fdee7c040a0a8dc618e777f1182 [id=e4250f6c594f49818692b819437cd1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.944634 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36dcd848d7454d8dae946f05b6e883a1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:47.944911 2053 leader_election.cc:343] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.945042 2053 leader_election.cc:304] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:47.946252 3997 raft_consensus.cc:2749] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:47.887506 2231 tablet_service.cc:1505] Processing CreateTablet for tablet d9c641bcc9c945dc934474f42828fb79 (DEFAULT_TABLE table=default.Table_bc05ce655d584c6480d446730692ceeb [id=1b3cd64d9f6b4eeca468545a71e1da73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.888018 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfd3e23d83a84648bd11cd258377aecb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:47.948549 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9c641bcc9c945dc934474f42828fb79. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:47.948761 2053 leader_election.cc:343] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:47.948882 2053 leader_election.cc:304] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:47.922811 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62d30d79a6e4432a8dbe40a104e87fa0" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:47.949393 2262 raft_consensus.cc:2468] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:47.923025 2227 tablet_service.cc:1505] Processing CreateTablet for tablet 19de884da8b64d8eaebeaae646664286 (DEFAULT_TABLE table=default.Table_2f7abb9f1aaf4a52b3c0066e12817faf [id=35816ac0236a4d6ca2786cca66603bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.927702 2377 ts_tablet_manager.cc:1929] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:47.950145 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19de884da8b64d8eaebeaae646664286. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.950224 2377 log.cc:1199] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/dba102838945453cbf44e1cfe505a52c
I20251026 02:56:47.950592 2377 ts_tablet_manager.cc:1950] T dba102838945453cbf44e1cfe505a52c P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:47.951339 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 906) took 6211 ms. Trace:
I20251026 02:56:47.951416 2377 rpcz_store.cc:276] 1026 02:56:41.739690 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.789540 (+ 49850us) service_pool.cc:225] Handling call
1026 02:56:47.670157 (+5880617us) ts_tablet_manager.cc:1151] Deleting tablet dba102838945453cbf44e1cfe505a52c
1026 02:56:47.670166 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.799715 (+129549us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.927606 (+127891us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.951314 (+ 23708us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5869925}
I20251026 02:56:47.952689 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 6667ddda96fb465ebdf54560f04c5555 (DEFAULT_TABLE table=default.Table_dbd53db2b8684c5ba304639f97a47556 [id=98e5d2bda13a42859a7d33fcd0cacfdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.953125 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6667ddda96fb465ebdf54560f04c5555. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.952153 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 952) took 4911 ms. Trace:
I20251026 02:56:47.954587 2372 rpcz_store.cc:276] 1026 02:56:43.040777 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.040840 (+ 63us) service_pool.cc:225] Handling call
1026 02:56:47.678462 (+4637622us) ts_tablet_manager.cc:1151] Deleting tablet 9a345fa5edbd48d2891b8b40b09cb3f8
1026 02:56:47.678474 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.779560 (+101086us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.935337 (+155777us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.952128 (+ 16791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4637420}
I20251026 02:56:47.955034 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 950) took 4914 ms. Trace:
I20251026 02:56:47.955155 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 9a345fa5edbd48d2891b8b40b09cb3f8 (table default.Table_3d25c519f16647ca962f3862c6cd4f98 [id=eca037eee4b54221b7c9771c390055c8]) successfully deleted
I20251026 02:56:47.955124 2372 rpcz_store.cc:276] 1026 02:56:43.040557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.040984 (+ 427us) service_pool.cc:225] Handling call
1026 02:56:47.954919 (+4913935us) ts_tablet_manager.cc:1151] Deleting tablet a6f9ae327aa4401baf66a737eaa93cf6
1026 02:56:47.954933 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.955018 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4913809}
I20251026 02:56:47.955471 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 954) took 4762 ms. Trace:
I20251026 02:56:47.955549 2372 rpcz_store.cc:276] 1026 02:56:43.193204 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.196437 (+ 3233us) service_pool.cc:225] Handling call
1026 02:56:47.955355 (+4758918us) ts_tablet_manager.cc:1151] Deleting tablet 19918cb3b7d0450f9de17026f8ea3295
1026 02:56:47.955367 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.955456 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4754267}
W20251026 02:56:47.955631 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a6f9ae327aa4401baf66a737eaa93cf6 because the tablet was not found. No further retry: Not found: Tablet not found: a6f9ae327aa4401baf66a737eaa93cf6
I20251026 02:56:47.955771 2372 tablet_replica.cc:333] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:47.955957 2372 raft_consensus.cc:2243] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:47.956115 2372 raft_consensus.cc:2272] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:47.956096 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 19918cb3b7d0450f9de17026f8ea3295 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19918cb3b7d0450f9de17026f8ea3295 already in progress: creating tablet
I20251026 02:56:47.956667 2372 ts_tablet_manager.cc:1916] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:47.957661 2377 tablet_replica.cc:333] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:47.958181 3997 raft_consensus.cc:2749] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:47.961791 2377 raft_consensus.cc:2243] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:47.962057 2377 raft_consensus.cc:2272] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:47.963059 2377 ts_tablet_manager.cc:1916] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:47.965932 2228 tablet_service.cc:1505] Processing CreateTablet for tablet fbb5e30694d2411f8c1932297837cf67 (DEFAULT_TABLE table=default.Table_089d38ef94b2431f926063d1eb24d294 [id=76f302ac2e2d4907b1984d19fa1ab1be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.966390 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet dba102838945453cbf44e1cfe505a52c (table default.Table_8ffce6cf6b2d44aa8a9459e834de1d80 [id=92310ee99b5c428c9773b0f7a5462a4d]) successfully deleted
I20251026 02:56:47.966511 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbb5e30694d2411f8c1932297837cf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.965926 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 58043f2dd48d4fffa613b3693e13dc81 (DEFAULT_TABLE table=default.Table_4c255180d55843908eaf691ea0b0a6ec [id=9c30114a0a1f4b96993cdc043f95fc20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.975359 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58043f2dd48d4fffa613b3693e13dc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.980274 2375 tablet_bootstrap.cc:492] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:47.980425 2375 ts_tablet_manager.cc:1403] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.109s user 0.004s sys 0.000s
I20251026 02:56:47.981120 2375 raft_consensus.cc:359] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.981303 2375 raft_consensus.cc:385] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.981359 2375 raft_consensus.cc:740] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.983974 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 102944a3ae83481089df88fc6b8388af (DEFAULT_TABLE table=default.Table_1697506af603436e82df1f54f1c42616 [id=0850387346f34dd99b7a7d0189567bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.984392 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 102944a3ae83481089df88fc6b8388af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.949800 1915 leader_election.cc:304] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:47.960387 2378 tablet_bootstrap.cc:492] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:47.988494 2378 ts_tablet_manager.cc:1403] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.312s user 0.004s sys 0.001s
I20251026 02:56:47.989250 2378 raft_consensus.cc:359] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.989418 2378 raft_consensus.cc:385] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:47.989464 2378 raft_consensus.cc:740] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:47.989651 2378 consensus_queue.cc:260] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.990074 2378 ts_tablet_manager.cc:1434] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 02:56:47.990347 2378 tablet_bootstrap.cc:492] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:47.990485 4095 raft_consensus.cc:2804] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:47.990541 4095 raft_consensus.cc:493] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:47.990592 4095 raft_consensus.cc:3060] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.993269 4095 raft_consensus.cc:515] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:47.993896 4095 leader_election.cc:290] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:47.994374 2094 tablet_service.cc:1505] Processing CreateTablet for tablet f5c835b08f504ae88274d05daa9113bc (DEFAULT_TABLE table=default.Table_98144cf085204d6ba89b8612ab94c4b5 [id=14d4d49f077949f7a12a3885d0009006]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:47.994676 1916 leader_election.cc:343] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 62d30d79a6e4432a8dbe40a104e87fa0
I20251026 02:56:47.994771 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c835b08f504ae88274d05daa9113bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:47.967216 1960 tablet_service.cc:1505] Processing CreateTablet for tablet fbb5e30694d2411f8c1932297837cf67 (DEFAULT_TABLE table=default.Table_089d38ef94b2431f926063d1eb24d294 [id=76f302ac2e2d4907b1984d19fa1ab1be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.968935 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 73f0b1443e2c4294a222e62433f3b435 (DEFAULT_TABLE table=default.Table_0982fce6dbaa41f69b6e80e254f245db [id=39a1785e03fd45b7a8ea0f14bf5a362a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:47.969794 4150 tablet.cc:1722] T 4db275efdd344817b11a2168675cf95e P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:47.996121 2376 log.cc:1199] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/5c6e3e47b3e7421e945cfbe671b06abe
I20251026 02:56:47.997201 2376 ts_tablet_manager.cc:1950] T 5c6e3e47b3e7421e945cfbe671b06abe P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:47.997879 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 916) took 4302 ms. Trace:
I20251026 02:56:47.998018 2376 rpcz_store.cc:276] 1026 02:56:43.695521 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.695723 (+ 202us) service_pool.cc:225] Handling call
1026 02:56:47.904191 (+4208468us) ts_tablet_manager.cc:1151] Deleting tablet 5c6e3e47b3e7421e945cfbe671b06abe
1026 02:56:47.904200 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:47.937392 (+ 33192us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.940441 (+ 3049us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:47.997826 (+ 57385us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4208068}
I20251026 02:56:47.998382 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62d30d79a6e4432a8dbe40a104e87fa0" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:47.998572 2262 raft_consensus.cc:3060] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:47.999279 2376 tablet_replica.cc:333] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:47.999760 2376 raft_consensus.cc:2243] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.000095 2376 raft_consensus.cc:2272] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.001439 2262 raft_consensus.cc:2468] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:48.001413 2376 ts_tablet_manager.cc:1916] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.002866 4148 tablet.cc:1722] T 4db275efdd344817b11a2168675cf95e P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:48.003422 2375 consensus_queue.cc:260] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.004422 1915 leader_election.cc:304] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:48.004712 4089 raft_consensus.cc:2804] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:48.004829 4089 raft_consensus.cc:697] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:48.005056 4089 consensus_queue.cc:237] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.005636 2096 tablet_service.cc:1505] Processing CreateTablet for tablet f2febe856ba04f549c66ea7d95f54482 (DEFAULT_TABLE table=default.Table_69e73d553b1b47f6b4494421985d7889 [id=b4210c3d42a64ee9b1c853aa98da6f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.006255 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2febe856ba04f549c66ea7d95f54482. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.010787 2375 ts_tablet_manager.cc:1434] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20251026 02:56:48.012626 4067 raft_consensus.cc:493] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:47.983587 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7847c12059f440cf83e6f1c7dc0f6808. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.016691 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f0b1443e2c4294a222e62433f3b435. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.020412 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 8869062eedda4bfe99b1793ae0353680 (DEFAULT_TABLE table=default.Table_ac6c5e4166e14e079228d171f79e964c [id=c8068196e1c74089abc3969544d314ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.020860 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8869062eedda4bfe99b1793ae0353680. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.026517 2378 tablet_bootstrap.cc:654] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.028466 4095 consensus_queue.cc:1048] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:48.029017 2375 tablet_bootstrap.cc:492] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:48.029685 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 73f0b1443e2c4294a222e62433f3b435 (DEFAULT_TABLE table=default.Table_0982fce6dbaa41f69b6e80e254f245db [id=39a1785e03fd45b7a8ea0f14bf5a362a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.030169 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f0b1443e2c4294a222e62433f3b435. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:48.035858 2195 consensus_peers.cc:597] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:48.036796 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbb5e30694d2411f8c1932297837cf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.039232 1943 tablet_service.cc:1505] Processing CreateTablet for tablet f5c835b08f504ae88274d05daa9113bc (DEFAULT_TABLE table=default.Table_98144cf085204d6ba89b8612ab94c4b5 [id=14d4d49f077949f7a12a3885d0009006]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.039434 1955 tablet_service.cc:1505] Processing CreateTablet for tablet f2febe856ba04f549c66ea7d95f54482 (DEFAULT_TABLE table=default.Table_69e73d553b1b47f6b4494421985d7889 [id=b4210c3d42a64ee9b1c853aa98da6f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.039604 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c835b08f504ae88274d05daa9113bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.039820 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2febe856ba04f549c66ea7d95f54482. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.040186 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 102944a3ae83481089df88fc6b8388af (DEFAULT_TABLE table=default.Table_1697506af603436e82df1f54f1c42616 [id=0850387346f34dd99b7a7d0189567bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.040304 2241 tablet_service.cc:1505] Processing CreateTablet for tablet f5c835b08f504ae88274d05daa9113bc (DEFAULT_TABLE table=default.Table_98144cf085204d6ba89b8612ab94c4b5 [id=14d4d49f077949f7a12a3885d0009006]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.040576 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 102944a3ae83481089df88fc6b8388af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.040745 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c835b08f504ae88274d05daa9113bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.050961 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 8869062eedda4bfe99b1793ae0353680 (DEFAULT_TABLE table=default.Table_ac6c5e4166e14e079228d171f79e964c [id=c8068196e1c74089abc3969544d314ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.051339 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8869062eedda4bfe99b1793ae0353680. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.015240 4067 raft_consensus.cc:515] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.053040 4067 leader_election.cc:290] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:48.053710 2195 leader_election.cc:343] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9a345fa5edbd48d2891b8b40b09cb3f8
I20251026 02:56:48.062163 2098 tablet_service.cc:1505] Processing CreateTablet for tablet a8749da375204fe0b3dcc95345ec995e (DEFAULT_TABLE table=default.Table_338870d29792464aa2ab8c8c2f5978ac [id=a3bc003739c141999f137843d4b24730]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.062614 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8749da375204fe0b3dcc95345ec995e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.069774 2372 ts_tablet_manager.cc:1929] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:48.070708 4167 raft_consensus.cc:493] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.070936 4167 raft_consensus.cc:515] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.071117 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a345fa5edbd48d2891b8b40b09cb3f8" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:48.071708 4167 leader_election.cc:290] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:48.072480 2194 leader_election.cc:343] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.072602 2194 leader_election.cc:304] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.072990 4003 raft_consensus.cc:2749] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.083021 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5228f482c0b4fcbbdf8ebc31bb62819" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:48.084254 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 5c6e3e47b3e7421e945cfbe671b06abe (table default.Table_653a3f85c24140cdb62a3e9f3ea95fd4 [id=a305da1e3a694b32aff05dbd6c58a6cb]) successfully deleted
I20251026 02:56:48.101150 2235 tablet_service.cc:1505] Processing CreateTablet for tablet f2febe856ba04f549c66ea7d95f54482 (DEFAULT_TABLE table=default.Table_69e73d553b1b47f6b4494421985d7889 [id=b4210c3d42a64ee9b1c853aa98da6f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.101198 2240 tablet_service.cc:1505] Processing CreateTablet for tablet a8749da375204fe0b3dcc95345ec995e (DEFAULT_TABLE table=default.Table_338870d29792464aa2ab8c8c2f5978ac [id=a3bc003739c141999f137843d4b24730]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.101603 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2febe856ba04f549c66ea7d95f54482. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.101708 1943 tablet_service.cc:1505] Processing CreateTablet for tablet a8749da375204fe0b3dcc95345ec995e (DEFAULT_TABLE table=default.Table_338870d29792464aa2ab8c8c2f5978ac [id=a3bc003739c141999f137843d4b24730]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.102049 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8749da375204fe0b3dcc95345ec995e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.104456 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8749da375204fe0b3dcc95345ec995e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.105185 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 102944a3ae83481089df88fc6b8388af (DEFAULT_TABLE table=default.Table_1697506af603436e82df1f54f1c42616 [id=0850387346f34dd99b7a7d0189567bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.105571 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 102944a3ae83481089df88fc6b8388af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.107251 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 03eb7f47578247a48a885489730d527e (DEFAULT_TABLE table=default.Table_e372c60e85634713bc49acb9d2fbee32 [id=1abf6f3514314cd2aec4fa6f13e4cda7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.107635 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03eb7f47578247a48a885489730d527e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.114198 2378 tablet_bootstrap.cc:492] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:48.114358 2378 ts_tablet_manager.cc:1403] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.124s user 0.001s sys 0.009s
I20251026 02:56:48.114959 2378 raft_consensus.cc:359] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.115134 2378 raft_consensus.cc:385] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.115183 2378 raft_consensus.cc:740] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.115406 2378 consensus_queue.cc:260] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.115878 2378 ts_tablet_manager.cc:1434] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20251026 02:56:48.116274 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 03eb7f47578247a48a885489730d527e (DEFAULT_TABLE table=default.Table_e372c60e85634713bc49acb9d2fbee32 [id=1abf6f3514314cd2aec4fa6f13e4cda7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.116696 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03eb7f47578247a48a885489730d527e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.120554 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 101ccbfd029c4087b12af6621f34a4bc (DEFAULT_TABLE table=default.Table_804ab36efead4908af699d1174155449 [id=405444369691404194082f51806c46be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.120956 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 101ccbfd029c4087b12af6621f34a4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.124538 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 101ccbfd029c4087b12af6621f34a4bc (DEFAULT_TABLE table=default.Table_804ab36efead4908af699d1174155449 [id=405444369691404194082f51806c46be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.124936 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 101ccbfd029c4087b12af6621f34a4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.128551 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 3183f909b0334d95b9ff9417c2954766 (DEFAULT_TABLE table=default.Table_d2755d308fc34447b8e72260cb3c0f76 [id=9340ac064cb1420c83c5abe892bf6ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.128932 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3183f909b0334d95b9ff9417c2954766. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.132560 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 3183f909b0334d95b9ff9417c2954766 (DEFAULT_TABLE table=default.Table_d2755d308fc34447b8e72260cb3c0f76 [id=9340ac064cb1420c83c5abe892bf6ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.132949 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3183f909b0334d95b9ff9417c2954766. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:48.133282 1629 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 52274.7us
W20251026 02:56:48.133952 1916 consensus_peers.cc:597] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f719c720cc474181a63786a3e248ab2c. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:48.145617 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 599a88ec0e76409c97ab62d6f55ebcd0
I20251026 02:56:48.146569 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 5a00bc04b3654f90af342986751344e6
I20251026 02:56:48.146952 1663 catalog_manager.cc:6719] ReplaceTablet: tablet 5a00bc04b3654f90af342986751344e6 deleted and replaced by tablet ee23940460a64d7ea1fe97758bb85fcb
I20251026 02:56:48.147467 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet c0e6905e5ac147e3bc3a0073816faf76
I20251026 02:56:48.148466 4150 tablet.cc:1722] T 23062e7292f44ad5ad9dfc22a401741c P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:48.148955 1654 catalog_manager.cc:5649] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:48.149483 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 386e1bd4635b4a7e8e7f249b577fd978 (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]). Expected version 2 got 0
I20251026 02:56:48.153328 2377 ts_tablet_manager.cc:1929] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.153437 2377 log.cc:1199] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/b1ca5b8379c94deab1130bfe8bdc87b0
I20251026 02:56:48.153753 2377 ts_tablet_manager.cc:1950] T b1ca5b8379c94deab1130bfe8bdc87b0 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:48.153950 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 086b40036f0c4c97bb95647f91ce7939
I20251026 02:56:48.154358 1655 catalog_manager.cc:6719] ReplaceTablet: tablet 086b40036f0c4c97bb95647f91ce7939 deleted and replaced by tablet 77d4998abe3a4a53a980b6d94b0021a9
I20251026 02:56:48.154595 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 913) took 6341 ms. Trace:
I20251026 02:56:48.154695 2377 rpcz_store.cc:276] 1026 02:56:41.813061 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:41.813411 (+ 350us) service_pool.cc:225] Handling call
1026 02:56:47.957613 (+6144202us) ts_tablet_manager.cc:1151] Deleting tablet b1ca5b8379c94deab1130bfe8bdc87b0
1026 02:56:47.957624 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.106887 (+149263us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.153210 (+ 46323us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.154568 (+ 1358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6138915}
I20251026 02:56:48.155143 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 918) took 5489 ms. Trace:
I20251026 02:56:48.155233 2377 rpcz_store.cc:276] 1026 02:56:42.665209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.665359 (+ 150us) service_pool.cc:225] Handling call
1026 02:56:48.155002 (+5489643us) ts_tablet_manager.cc:1151] Deleting tablet 5b8e1d79d2a247dba25540daf6b81bf0
1026 02:56:48.155013 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.155126 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5485426}
I20251026 02:56:48.158655 2372 log.cc:1199] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/95c9fbae62db4850b0f02a2443a3432b
I20251026 02:56:48.159104 2372 ts_tablet_manager.cc:1950] T 95c9fbae62db4850b0f02a2443a3432b P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:48.072803 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 03eb7f47578247a48a885489730d527e (DEFAULT_TABLE table=default.Table_e372c60e85634713bc49acb9d2fbee32 [id=1abf6f3514314cd2aec4fa6f13e4cda7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.160144 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03eb7f47578247a48a885489730d527e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.073117 2375 tablet_bootstrap.cc:654] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.163707 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 962) took 4936 ms. Trace:
I20251026 02:56:48.081748 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 101ccbfd029c4087b12af6621f34a4bc (DEFAULT_TABLE table=default.Table_804ab36efead4908af699d1174155449 [id=405444369691404194082f51806c46be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.163846 2372 rpcz_store.cc:276] 1026 02:56:43.226875 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.226927 (+ 52us) service_pool.cc:225] Handling call
1026 02:56:47.955731 (+4728804us) ts_tablet_manager.cc:1151] Deleting tablet 95c9fbae62db4850b0f02a2443a3432b
1026 02:56:47.955739 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.066924 (+111185us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.069687 (+ 2763us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.163684 (+ 93997us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4728624}
I20251026 02:56:48.164237 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 101ccbfd029c4087b12af6621f34a4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.164397 2372 tablet_replica.cc:333] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:48.164615 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 95c9fbae62db4850b0f02a2443a3432b (table default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]) successfully deleted
I20251026 02:56:48.165243 2372 raft_consensus.cc:2243] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.165894 2372 raft_consensus.cc:2272] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.166742 2372 ts_tablet_manager.cc:1916] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.169179 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 921) took 5503 ms. Trace:
I20251026 02:56:48.170297 2376 ts_tablet_manager.cc:1929] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:48.170481 2376 log.cc:1199] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3dbe8002df6c4b64b8ec8260ca6b9cf5
I20251026 02:56:48.170596 2377 rpcz_store.cc:276] 1026 02:56:42.665492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.665535 (+ 43us) service_pool.cc:225] Handling call
1026 02:56:48.169042 (+5503507us) ts_tablet_manager.cc:1151] Deleting tablet 663076e362c44837beb322aa965cde42
1026 02:56:48.169054 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.169161 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5498975}
I20251026 02:56:48.171161 2376 ts_tablet_manager.cc:1950] T 3dbe8002df6c4b64b8ec8260ca6b9cf5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:48.171687 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 918) took 4440 ms. Trace:
I20251026 02:56:48.172837 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 086b40036f0c4c97bb95647f91ce7939 with delete_type TABLET_DATA_DELETED (replaced by tablet 77d4998abe3a4a53a980b6d94b0021a9) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.095358 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 3183f909b0334d95b9ff9417c2954766 (DEFAULT_TABLE table=default.Table_d2755d308fc34447b8e72260cb3c0f76 [id=9340ac064cb1420c83c5abe892bf6ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.178580 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3183f909b0334d95b9ff9417c2954766. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:48.101801 2053 consensus_peers.cc:597] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:48.180315 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 78553.2us
I20251026 02:56:48.150544 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 599a88ec0e76409c97ab62d6f55ebcd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.150713 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a00bc04b3654f90af342986751344e6 with delete_type TABLET_DATA_DELETED (replaced by tablet ee23940460a64d7ea1fe97758bb85fcb) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.150833 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet c0e6905e5ac147e3bc3a0073816faf76 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.072329 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5228f482c0b4fcbbdf8ebc31bb62819" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:48.182888 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet b1ca5b8379c94deab1130bfe8bdc87b0 (table default.Table_85c523c971b34e68bed4afa5d1aac450 [id=32c06e4782a8441c8d1614e81277b52e]) successfully deleted
I20251026 02:56:48.183259 4162 consensus_queue.cc:1048] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:48.183465 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 5b8e1d79d2a247dba25540daf6b81bf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b8e1d79d2a247dba25540daf6b81bf0 already in progress: creating tablet
W20251026 02:56:48.183616 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 663076e362c44837beb322aa965cde42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 663076e362c44837beb322aa965cde42 already in progress: creating tablet
I20251026 02:56:48.186488 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 919) took 5521 ms. Trace:
I20251026 02:56:48.187144 2377 rpcz_store.cc:276] 1026 02:56:42.665307 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.665529 (+ 222us) service_pool.cc:225] Handling call
1026 02:56:48.171369 (+5505840us) ts_tablet_manager.cc:1151] Deleting tablet 2f197526e7d34f849dbb4f28d8d414cf
1026 02:56:48.186322 (+ 14953us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.186461 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5501116}
I20251026 02:56:48.189385 2116 raft_consensus.cc:2468] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:48.190340 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 8869062eedda4bfe99b1793ae0353680 (DEFAULT_TABLE table=default.Table_ac6c5e4166e14e079228d171f79e964c [id=c8068196e1c74089abc3969544d314ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.190717 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8869062eedda4bfe99b1793ae0353680. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.189735 2225 tablet_service.cc:1552] Processing DeleteTablet for tablet 599a88ec0e76409c97ab62d6f55ebcd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.191236 2225 tablet_service.cc:1552] Processing DeleteTablet for tablet 086b40036f0c4c97bb95647f91ce7939 with delete_type TABLET_DATA_DELETED (replaced by tablet 77d4998abe3a4a53a980b6d94b0021a9) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.189800 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a00bc04b3654f90af342986751344e6 with delete_type TABLET_DATA_DELETED (replaced by tablet ee23940460a64d7ea1fe97758bb85fcb) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.189905 2236 tablet_service.cc:1552] Processing DeleteTablet for tablet c0e6905e5ac147e3bc3a0073816faf76 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.191641 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 922) took 5525 ms. Trace:
I20251026 02:56:48.182392 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet c0e6905e5ac147e3bc3a0073816faf76 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.182896 1916 leader_election.cc:304] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:48.196314 2376 rpcz_store.cc:276] 1026 02:56:43.731596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.731664 (+ 68us) service_pool.cc:225] Handling call
1026 02:56:47.998898 (+4267234us) ts_tablet_manager.cc:1151] Deleting tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5
1026 02:56:47.998946 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.132156 (+133210us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.158393 (+ 26237us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.171667 (+ 13274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4266937}
I20251026 02:56:48.196882 4098 raft_consensus.cc:493] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.197108 4098 raft_consensus.cc:515] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.198447 2376 tablet_replica.cc:333] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:48.198813 4098 leader_election.cc:290] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.184792 2378 tablet_bootstrap.cc:492] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:48.201426 2378 tablet_bootstrap.cc:654] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.192556 2256 raft_consensus.cc:1275] T 2fd8f02126b24285bbfa575540ba0f09 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Refusing update from remote peer 0b6b0a6c4e144982b67980118cddba48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 02:56:48.202028 4098 raft_consensus.cc:493] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.202239 4098 raft_consensus.cc:515] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.190696 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a00bc04b3654f90af342986751344e6 with delete_type TABLET_DATA_DELETED (replaced by tablet ee23940460a64d7ea1fe97758bb85fcb) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.196599 4167 raft_consensus.cc:2749] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.183020 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 086b40036f0c4c97bb95647f91ce7939 with delete_type TABLET_DATA_DELETED (replaced by tablet 77d4998abe3a4a53a980b6d94b0021a9) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.190354 2195 leader_election.cc:304] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.199093 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b77bb6b4722449c9699b548a9ef6314" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:48.201565 2377 rpcz_store.cc:276] 1026 02:56:42.665749 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.665795 (+ 46us) service_pool.cc:225] Handling call
1026 02:56:48.191519 (+5525724us) ts_tablet_manager.cc:1151] Deleting tablet 8977f824ff374d05b4ed9d88c67329f9
1026 02:56:48.191530 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.191626 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5521139}
I20251026 02:56:48.203217 4009 raft_consensus.cc:2804] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:48.203325 4009 raft_consensus.cc:697] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:48.203413 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 923) took 5537 ms. Trace:
I20251026 02:56:48.203487 2377 rpcz_store.cc:276] 1026 02:56:42.665986 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.666043 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:48.203295 (+5537252us) ts_tablet_manager.cc:1151] Deleting tablet a4f4d7f87916425999574fd5b9f2fdf7
1026 02:56:48.203305 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.203400 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5532850}
I20251026 02:56:48.203563 4009 consensus_queue.cc:237] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.203774 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 924) took 5537 ms. Trace:
I20251026 02:56:48.203835 2377 rpcz_store.cc:276] 1026 02:56:42.666084 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.666135 (+ 51us) service_pool.cc:225] Handling call
1026 02:56:48.203683 (+5537548us) ts_tablet_manager.cc:1151] Deleting tablet ac82c14e7dcb448b845d998249e2b339
1026 02:56:48.203692 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.203761 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5533173}
I20251026 02:56:48.204077 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 925) took 5537 ms. Trace:
I20251026 02:56:48.204134 2377 rpcz_store.cc:276] 1026 02:56:42.666209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:42.666265 (+ 56us) service_pool.cc:225] Handling call
1026 02:56:48.203995 (+5537730us) ts_tablet_manager.cc:1151] Deleting tablet b8fe7825fa8d46669772cf84bc4b3afc
1026 02:56:48.204003 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.204064 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5533444}
I20251026 02:56:48.204386 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 944) took 5033 ms. Trace:
I20251026 02:56:48.204414 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b5e61afe90e4bc4a2897de21e0b3919" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:48.204476 2377 rpcz_store.cc:276] 1026 02:56:43.171311 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.171367 (+ 56us) service_pool.cc:225] Handling call
1026 02:56:48.204317 (+5032950us) ts_tablet_manager.cc:1151] Deleting tablet 1d7ffdaeec2240dcbe4cafdb96073a7d
1026 02:56:48.204324 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.204372 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5032750}
I20251026 02:56:48.204586 1982 raft_consensus.cc:3060] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.204684 2377 tablet_replica.cc:333] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:48.204931 2377 raft_consensus.cc:2243] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:48.207404 2053 leader_election.cc:343] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.207726 4162 consensus_queue.cc:1048] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:48.208240 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b77bb6b4722449c9699b548a9ef6314" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:48.209630 2377 raft_consensus.cc:2272] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.210685 2377 ts_tablet_manager.cc:1916] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:48.210851 2053 leader_election.cc:343] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.211402 4098 leader_election.cc:290] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.184502 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 599a88ec0e76409c97ab62d6f55ebcd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.213367 2375 tablet_bootstrap.cc:492] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:48.213537 2375 ts_tablet_manager.cc:1403] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.185s user 0.010s sys 0.001s
I20251026 02:56:48.214430 2375 raft_consensus.cc:359] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.214663 2375 raft_consensus.cc:385] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.214725 2375 raft_consensus.cc:740] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.214983 2375 consensus_queue.cc:260] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.215566 2375 ts_tablet_manager.cc:1434] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:48.215873 2375 tablet_bootstrap.cc:492] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:48.215960 2053 leader_election.cc:343] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.216041 2053 leader_election.cc:304] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.216408 4098 raft_consensus.cc:2749] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:48.219907 2194 consensus_peers.cc:597] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:48.221097 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b5e61afe90e4bc4a2897de21e0b3919" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:48.222044 2053 leader_election.cc:343] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.222211 2053 leader_election.cc:304] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.222460 3997 raft_consensus.cc:2749] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.225906 2375 tablet_bootstrap.cc:654] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.231364 2376 raft_consensus.cc:2243] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.231724 2376 raft_consensus.cc:2272] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.232517 2376 ts_tablet_manager.cc:1916] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:48.242827 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 2f197526e7d34f849dbb4f28d8d414cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f197526e7d34f849dbb4f28d8d414cf already in progress: creating tablet
W20251026 02:56:48.243227 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 8977f824ff374d05b4ed9d88c67329f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8977f824ff374d05b4ed9d88c67329f9 already in progress: creating tablet
W20251026 02:56:48.243546 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ac82c14e7dcb448b845d998249e2b339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac82c14e7dcb448b845d998249e2b339 already in progress: creating tablet
W20251026 02:56:48.243736 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet b8fe7825fa8d46669772cf84bc4b3afc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8fe7825fa8d46669772cf84bc4b3afc already in progress: creating tablet
W20251026 02:56:48.243913 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 1d7ffdaeec2240dcbe4cafdb96073a7d because the tablet was not found. No further retry: Not found: Tablet not found: 1d7ffdaeec2240dcbe4cafdb96073a7d
I20251026 02:56:48.244261 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3dbe8002df6c4b64b8ec8260ca6b9cf5 (table default.Table_2dfe8bcc7f1e4933a4174bdab19cffd5 [id=65d576adb71949cc97666624935adbf3]) successfully deleted
I20251026 02:56:48.247848 2381 tablet_bootstrap.cc:492] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:48.250411 2381 tablet_bootstrap.cc:654] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.254172 2375 tablet_bootstrap.cc:492] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:48.254326 2375 ts_tablet_manager.cc:1403] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20251026 02:56:48.255142 2375 raft_consensus.cc:359] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.255333 2375 raft_consensus.cc:385] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.255390 2375 raft_consensus.cc:740] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
W20251026 02:56:48.259155 1660 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.2:34337 (request call id 158) took 884 ms (client timeout 1000 ms). Trace:
W20251026 02:56:48.259344 1660 rpcz_store.cc:269] 1026 02:56:47.374711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.395085 (+ 20374us) service_pool.cc:225] Handling call
1026 02:56:47.397768 (+ 2683us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x616000041254. stack: 00007f54638f59f6 00007f54638f57b0 00007f54628d13ad 0000000000983f04 0000000000983ecd 00007f5472bf36bd 00007f5472bf37b3 00007f546f7be441 00007f54774749dd 00007f54775fbaa8 00007f546e114ca6 00007f546e1149d5 00007f546a89127e 00007f546a89015e 00007f546a896395 00007f546a896cbf
1026 02:56:48.259132 (+861364us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:47.857525 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:47.857617 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:47.857622 (+ 5us) write_op.cc:435] Acquired schema lock
1026 02:56:47.857624 (+ 2us) tablet.cc:660] Decoding operations
1026 02:56:47.857694 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:47.857703 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:47.857710 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
1026 02:56:47.857748 (+ 38us) tablet.cc:699] Row locks acquired
1026 02:56:47.857750 (+ 2us) write_op.cc:260] PREPARE: finished
1026 02:56:47.857772 (+ 22us) write_op.cc:270] Start()
1026 02:56:47.857792 (+ 20us) write_op.cc:276] Timestamp: P: 1761447407857717 usec, L: 0
1026 02:56:47.857795 (+ 3us) op_driver.cc:348] REPLICATION: starting
1026 02:56:47.857902 (+ 107us) log.cc:844] Serialized 527 byte log entry
1026 02:56:48.256337 (+398435us) op_driver.cc:464] REPLICATION: finished
1026 02:56:48.256492 (+ 155us) write_op.cc:301] APPLY: starting
1026 02:56:48.256522 (+ 30us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:48.256565 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:48.256609 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:48.256670 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:48.256702 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:48.256708 (+ 6us) write_op.cc:312] APPLY: finished
1026 02:56:48.256790 (+ 82us) log.cc:844] Serialized 23 byte log entry
1026 02:56:48.256846 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:48.256864 (+ 18us) write_op.cc:454] Released schema lock
1026 02:56:48.256870 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:48.256932 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:48.256936 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":4584192,"child_traces":[["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":434,"apply.run_wall_time_us":434,"num_ops":1,"prepare.queue_time_us":142882,"prepare.run_cpu_time_us":443,"prepare.run_wall_time_us":442,"replication_time_us":398508}]]}
I20251026 02:56:48.260879 2372 ts_tablet_manager.cc:1929] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:48.260984 2372 log.cc:1199] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/868dbbfce40c4dd6b36cc220966b1986
I20251026 02:56:48.261261 2372 ts_tablet_manager.cc:1950] T 868dbbfce40c4dd6b36cc220966b1986 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:48.261673 2375 consensus_queue.cc:260] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.262946 2378 tablet_bootstrap.cc:492] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:48.263093 2378 ts_tablet_manager.cc:1403] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.007s
I20251026 02:56:48.263741 2378 raft_consensus.cc:359] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.263851 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 6699b50d87964f6a89c365171253ad25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.263908 2378 raft_consensus.cc:385] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.263962 2378 raft_consensus.cc:740] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.264180 2378 consensus_queue.cc:260] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.264696 2378 ts_tablet_manager.cc:1434] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 02:56:48.264950 2378 tablet_bootstrap.cc:492] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:48.265722 2377 ts_tablet_manager.cc:1929] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.266067 2375 ts_tablet_manager.cc:1434] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251026 02:56:48.266407 2378 tablet_bootstrap.cc:654] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.268178 2375 tablet_bootstrap.cc:492] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:48.269218 1981 raft_consensus.cc:3060] T f5228f482c0b4fcbbdf8ebc31bb62819 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.270429 1980 raft_consensus.cc:1275] T 2fd8f02126b24285bbfa575540ba0f09 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Refusing update from remote peer 0b6b0a6c4e144982b67980118cddba48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 02:56:48.270941 4098 consensus_queue.cc:1048] T 2fd8f02126b24285bbfa575540ba0f09 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:48.265798 4067 raft_consensus.cc:493] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.271765 4067 raft_consensus.cc:515] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.272574 4067 leader_election.cc:290] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.272790 2377 log.cc:1199] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/d212170f056a44dfb15cfcb7f9fbae2f
I20251026 02:56:48.276393 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a00bc04b3654f90af342986751344e6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:48.276861 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a00bc04b3654f90af342986751344e6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:48.277077 2110 raft_consensus.cc:2468] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:48.278725 2375 tablet_bootstrap.cc:654] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:48.277599 2194 leader_election.cc:343] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.273480 2377 ts_tablet_manager.cc:1950] T d212170f056a44dfb15cfcb7f9fbae2f P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:48.279662 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8869062eedda4bfe99b1793ae0353680
I20251026 02:56:48.281225 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 946) took 5109 ms. Trace:
I20251026 02:56:48.281342 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 8869062eedda4bfe99b1793ae0353680 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.282312 2377 rpcz_store.cc:276] 1026 02:56:43.171488 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.171687 (+ 199us) service_pool.cc:225] Handling call
1026 02:56:48.204646 (+5032959us) ts_tablet_manager.cc:1151] Deleting tablet d212170f056a44dfb15cfcb7f9fbae2f
1026 02:56:48.204653 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.249531 (+ 44878us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.265577 (+ 16046us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.281205 (+ 15628us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5032843}
I20251026 02:56:48.282621 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 4ed205c1800243ed9830d720ac85de2f
I20251026 02:56:48.283895 2377 tablet_replica.cc:333] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:48.284137 2377 raft_consensus.cc:2243] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:48.284569 2377 raft_consensus.cc:2272] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.284858 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ed205c1800243ed9830d720ac85de2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.285145 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 8869062eedda4bfe99b1793ae0353680 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.285216 2377 ts_tablet_manager.cc:1916] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.287000 2195 leader_election.cc:304] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.287454 4069 raft_consensus.cc:2804] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:48.287564 4069 raft_consensus.cc:493] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:48.287640 4069 raft_consensus.cc:3060] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.287819 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet 8869062eedda4bfe99b1793ae0353680 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.290648 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_63a0dc3ab9a14e37b44c2d6c776ca543"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:48.291271 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63a0dc3ab9a14e37b44c2d6c776ca543 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.291926 4069 raft_consensus.cc:515] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.292833 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e0fa37c306924b8ea7a9cd90c7d94b38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:48.293362 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0fa37c306924b8ea7a9cd90c7d94b38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.294554 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_03d048a39e0548078a18eb96c4830780"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:48.295011 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03d048a39e0548078a18eb96c4830780 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.295027 4069 leader_election.cc:290] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.295644 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a00bc04b3654f90af342986751344e6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:48.296326 2194 leader_election.cc:343] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.296391 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f1f3e2329722434cae69c91816fab475"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:48.296809 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1f3e2329722434cae69c91816fab475 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.297854 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c6919d91bdbd4df3965482f54a8c703e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:48.298271 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6919d91bdbd4df3965482f54a8c703e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.300956 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_472acbcc3a244422abe7a7477c65c87a" } new_table_name: "default.Table_cfdd28a8649a417b96cc700418d39b3f" modify_external_catalogs: true
I20251026 02:56:48.302223 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7d70886e009247ba950d471827c82385"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:48.302642 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d70886e009247ba950d471827c82385 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.303519 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet d212170f056a44dfb15cfcb7f9fbae2f (table default.Table_7667f4f99bf744579f1dc2b2d0f469eb [id=822eea61b89846f4828511c58a9da4c8]) successfully deleted
I20251026 02:56:48.303900 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ed205c1800243ed9830d720ac85de2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.280340 2381 tablet_bootstrap.cc:492] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:48.304478 2381 ts_tablet_manager.cc:1403] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.001s
I20251026 02:56:48.305469 2381 raft_consensus.cc:359] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.305706 2381 raft_consensus.cc:385] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.305804 2381 raft_consensus.cc:740] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.306046 2381 consensus_queue.cc:260] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.306725 2381 ts_tablet_manager.cc:1434] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:48.307052 2381 tablet_bootstrap.cc:492] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:48.309296 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 964) took 5065 ms. Trace:
I20251026 02:56:48.309468 2372 rpcz_store.cc:276] 1026 02:56:43.243456 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.260150 (+ 16694us) service_pool.cc:225] Handling call
1026 02:56:48.164355 (+4904205us) ts_tablet_manager.cc:1151] Deleting tablet 868dbbfce40c4dd6b36cc220966b1986
1026 02:56:48.164365 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.239311 (+ 74946us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.260773 (+ 21462us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.309265 (+ 48492us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4875487}
I20251026 02:56:48.310122 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 868dbbfce40c4dd6b36cc220966b1986 (table default.Table_c062fd73dfee44db8acf3124482a05ed [id=6ed9ef4d13484e92a0b292ed4085cea0]) successfully deleted
I20251026 02:56:48.310714 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ed205c1800243ed9830d720ac85de2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:47 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.321260 2381 tablet_bootstrap.cc:654] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.322536 2376 ts_tablet_manager.cc:1929] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.322808 2376 log.cc:1199] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/882fe81da21344ed99ea735fa79f73c0
I20251026 02:56:48.323572 2376 ts_tablet_manager.cc:1950] T 882fe81da21344ed99ea735fa79f73c0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:48.323875 2378 tablet_bootstrap.cc:492] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:48.324012 2378 ts_tablet_manager.cc:1403] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.059s user 0.000s sys 0.003s
I20251026 02:56:48.324535 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 919) took 4287 ms. Trace:
I20251026 02:56:48.330834 2381 tablet_bootstrap.cc:492] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:48.331117 2381 ts_tablet_manager.cc:1403] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20251026 02:56:48.324759 2378 raft_consensus.cc:359] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.332336 2378 raft_consensus.cc:385] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.332445 2378 raft_consensus.cc:740] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.332670 2378 consensus_queue.cc:260] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.333180 2378 ts_tablet_manager.cc:1434] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251026 02:56:48.333453 2378 tablet_bootstrap.cc:492] T 4b5e61afe90e4bc4a2897de21e0b3919 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:48.335228 2378 tablet_bootstrap.cc:654] T 4b5e61afe90e4bc4a2897de21e0b3919 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.336616 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a00bc04b3654f90af342986751344e6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:48.336892 2110 raft_consensus.cc:3060] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.336999 2372 tablet_replica.cc:333] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:48.337179 2372 raft_consensus.cc:2243] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:48.337509 2372 raft_consensus.cc:2272] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.345984 2372 ts_tablet_manager.cc:1916] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.351076 1624 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu pid 1700
I20251026 02:56:48.352473 1662 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 4db275efdd344817b11a2168675cf95e (table default.Table_c30cfa52b6524323bda478613d74178b [id=1fd74186ea10498599435dccd7330d25]). Expected version 1 got 0
I20251026 02:56:48.352797 1662 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 386e1bd4635b4a7e8e7f249b577fd978 (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]). Expected version 2 got 0
I20251026 02:56:48.353003 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a923651b1a2746818e0bbbbaf1826bb3" } new_table_name: "default.Table_1103bb37accf4daf87c2bc9427de7d43" modify_external_catalogs: true
I20251026 02:56:48.354821 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_62dd15084aac45f585bca051a5e7cfe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:48.355211 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3a75d3c1b8304d1bbcc19c12111f24dd" } new_table_name: "default.Table_3778c4a74e2f42e49c437a2aa3c5f22b" modify_external_catalogs: true
W20251026 02:56:48.355319 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62dd15084aac45f585bca051a5e7cfe5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.357125 1764 generic_service.cc:196] Checking for leaks (request via RPC)
I20251026 02:56:48.358880 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 62d30d79a6e4432a8dbe40a104e87fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.358880 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c7efb70c0804c149f36440a4a9e6ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.359021 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_dc22b2aa5fe64641b34fa498992594cc" } new_table_name: "default.Table_5d42dcc8c010466588247bd64b4b0b42" modify_external_catalogs: true
W20251026 02:56:48.352537 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fba4c45dc0448fc95469df5be374809 consensus state has no leader
I20251026 02:56:48.360767 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_30a03eb811714167a1169b464b284eef" } modify_external_catalogs: true
I20251026 02:56:48.360908 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_30a03eb811714167a1169b464b284eef" } modify_external_catalogs: true
I20251026 02:56:48.361588 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4e171e396e77413bb4d15fb2bcf9b0e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:48.362073 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e171e396e77413bb4d15fb2bcf9b0e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.364132 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1697506af603436e82df1f54f1c42616" } new_table_name: "default.Table_09e3951452224d23b58479544e06f857" modify_external_catalogs: true
I20251026 02:56:48.367447 1660 master_service.cc:954] ReplaceTablet: received request to replace tablet abf58185766448d1a7c4f3d304b0c5b1 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:48.374351 2381 raft_consensus.cc:359] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.375269 2381 raft_consensus.cc:385] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.332285 2376 rpcz_store.cc:276] 1026 02:56:44.036674 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.036724 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:48.198379 (+4161655us) ts_tablet_manager.cc:1151] Deleting tablet 882fe81da21344ed99ea735fa79f73c0
1026 02:56:48.198395 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.271086 (+ 72691us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.313230 (+ 42144us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.324515 (+ 11285us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4156355}
I20251026 02:56:48.375653 2381 raft_consensus.cc:740] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.376590 2381 consensus_queue.cc:260] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.377236 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 882fe81da21344ed99ea735fa79f73c0 (table default.Table_18e8723148dd46e6a56e5716a15d01bc [id=156e8ddc92b1471e8dce064bcb23306b]) successfully deleted
I20251026 02:56:48.377823 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 920) took 4226 ms. Trace:
I20251026 02:56:48.378852 2381 ts_tablet_manager.cc:1434] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.047s user 0.001s sys 0.000s
I20251026 02:56:48.379048 2376 rpcz_store.cc:276] 1026 02:56:44.151373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.152082 (+ 709us) service_pool.cc:225] Handling call
1026 02:56:48.377651 (+4225569us) ts_tablet_manager.cc:1151] Deleting tablet 0afb6f04f5ac463780fe8f522bf2fe23
1026 02:56:48.377667 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.377799 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4225357}
W20251026 02:56:48.379755 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0afb6f04f5ac463780fe8f522bf2fe23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0afb6f04f5ac463780fe8f522bf2fe23 already in progress: creating tablet
I20251026 02:56:48.380336 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 921) took 4213 ms. Trace:
I20251026 02:56:48.396335 2110 raft_consensus.cc:2468] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:48.396874 2195 leader_election.cc:304] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.404407 2372 ts_tablet_manager.cc:1929] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.404589 4069 raft_consensus.cc:2804] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:48.404722 4069 raft_consensus.cc:697] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:48.404976 4069 consensus_queue.cc:237] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.406252 3391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57034 (request call id 2) took 2284 ms. Trace:
I20251026 02:56:48.408875 2372 log.cc:1199] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/41b7b0c36ba748f0a1565892dbe98a78
I20251026 02:56:48.409665 2372 ts_tablet_manager.cc:1950] T 41b7b0c36ba748f0a1565892dbe98a78 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:48.410516 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 967) took 5121 ms. Trace:
I20251026 02:56:48.410615 2372 rpcz_store.cc:276] 1026 02:56:43.288869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.289137 (+ 268us) service_pool.cc:225] Handling call
1026 02:56:48.309949 (+5020812us) ts_tablet_manager.cc:1151] Deleting tablet 41b7b0c36ba748f0a1565892dbe98a78
1026 02:56:48.336946 (+ 26997us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.363567 (+ 26621us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.404272 (+ 40705us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.410483 (+ 6211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5020624}
I20251026 02:56:48.410878 2381 tablet_bootstrap.cc:492] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:48.411016 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 965) took 5165 ms. Trace:
I20251026 02:56:48.411303 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 41b7b0c36ba748f0a1565892dbe98a78 (table default.Table_292ad60362294da0976fd8fe6b180c0a [id=4651ecfc8aed4daf99de5a271e8afcda]) successfully deleted
I20251026 02:56:48.406507 4083 raft_consensus.cc:493] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 209968db4dee47c49497a66c3ee750b7)
I20251026 02:56:48.412106 4083 raft_consensus.cc:515] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:48.412295 2376 rpcz_store.cc:276] 1026 02:56:44.167263 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.167320 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:48.380174 (+4212854us) ts_tablet_manager.cc:1151] Deleting tablet 146d6f13b9ef47f29711a102791bab5a
1026 02:56:48.380183 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.380319 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4208037}
W20251026 02:56:48.412853 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 146d6f13b9ef47f29711a102791bab5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 146d6f13b9ef47f29711a102791bab5a already in progress: creating tablet
I20251026 02:56:48.378834 2378 tablet_bootstrap.cc:492] T 4b5e61afe90e4bc4a2897de21e0b3919 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:48.413201 2378 ts_tablet_manager.cc:1403] T 4b5e61afe90e4bc4a2897de21e0b3919 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.004s
I20251026 02:56:48.413947 3357 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:35202 (request call id 0) took 1801 ms. Trace:
I20251026 02:56:48.414064 3357 rpcz_store.cc:276] 1026 02:56:46.612817 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.612870 (+ 53us) service_pool.cc:225] Handling call
1026 02:56:48.409122 (+1796252us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x616000070954. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 00007fa0a6b2e6bd 00007fa0a6b2e7b3 00007fa0a36f9441 00007fa0ab3af9dd 00007fa0ab544965 00007fa0a2064747 00007fa0a2064475 00007fa09e7cc27e 00007fa09e7cb15e 00007fa09e7d1395 00007fa09e7d1cbf
1026 02:56:48.413660 (+ 4538us) spinlock_profiling.cc:243] Waited 3.86 ms on lock 0x616000070954. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fa0a36f8945 00007fa0ab32a095 00007fa0ab544cf0 00007fa0a2064747 00007fa0a2064475 00007fa09e7cc27e 00007fa09e7cb15e 00007fa09e7d1395 00007fa09e7d1cbf 00007fa09e7d1ae8
1026 02:56:48.413882 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4133075200}
W20251026 02:56:48.414832 3363 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.3:53887 (request call id 66) took 2271 ms (client timeout 1000 ms). Trace:
W20251026 02:56:48.414950 3363 rpcz_store.cc:269] 1026 02:56:46.142971 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.143037 (+ 66us) service_pool.cc:225] Handling call
1026 02:56:48.413339 (+2270302us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x616000070954. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 00007fa0a6b2e6bd 00007fa0a6b2e7b3 00007fa0a36f9441 00007fa0ab3af9dd 00007fa0ab536aa8 00007fa0a204fca6 00007fa0a204f9d5 00007fa09e7cc27e 00007fa09e7cb15e 00007fa09e7d1395 00007fa09e7d1cbf
1026 02:56:48.414774 (+ 1435us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5218253184}
I20251026 02:56:48.418345 3358 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:57034 (request call id 3) took 1989 ms. Trace:
I20251026 02:56:48.418445 3358 rpcz_store.cc:276] 1026 02:56:46.428628 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.428677 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:48.409790 (+1981113us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x616000070954. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 00007fa0a6b2e6bd 00007fa0a6b2e7b3 00007fa0a36f9441 00007fa0ab3af9dd 00007fa0ab544965 00007fa0a2064747 00007fa0a2064475 00007fa09e7cc27e 00007fa09e7cb15e 00007fa09e7d1395 00007fa09e7d1cbf
1026 02:56:48.418212 (+ 8422us) spinlock_profiling.cc:243] Waited 7.95 ms on lock 0x616000070954. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fa0a36f8945 00007fa0ab32a095 00007fa0ab544cf0 00007fa0a2064747 00007fa0a2064475 00007fa09e7cc27e 00007fa09e7cb15e 00007fa09e7d1395 00007fa09e7d1cbf 00007fa09e7d1ae8
1026 02:56:48.418286 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4565892224}
I20251026 02:56:48.419021 1673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 4 candidate_status { last_received { term: 3 index: 1293 } } ignore_live_leader: false dest_uuid: "209968db4dee47c49497a66c3ee750b7" is_pre_election: true
I20251026 02:56:48.411084 2372 rpcz_store.cc:276] 1026 02:56:43.245409 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.261079 (+ 15670us) service_pool.cc:225] Handling call
1026 02:56:48.410887 (+5149808us) ts_tablet_manager.cc:1151] Deleting tablet 9b9dc0825e0844f68d5afcc201db737c
1026 02:56:48.410899 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.411000 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5120784}
I20251026 02:56:48.419994 2372 tablet_replica.cc:333] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:48.420188 2372 raft_consensus.cc:2243] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:48.413115 4083 leader_election.cc:290] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 209968db4dee47c49497a66c3ee750b7 (127.1.150.62:44525), 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655)
W20251026 02:56:48.408547 3378 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.1:42831 (request call id 93) took 2243 ms (client timeout 1000 ms). Trace:
W20251026 02:56:48.420680 3378 rpcz_store.cc:269] 1026 02:56:46.164552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.164610 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:48.406654 (+2242044us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x616000070954. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 00007fa0a6b2e6bd 00007fa0a6b2e7b3 00007fa0a36f9441 00007fa0ab3af9dd 00007fa0ab536aa8 00007fa0a204fca6 00007fa0a204f9d5 00007fa09e7cc27e 00007fa09e7cb15e 00007fa09e7d1395 00007fa09e7d1cbf
1026 02:56:48.408439 (+ 1785us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5153597824}
W20251026 02:56:48.421906 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9b9dc0825e0844f68d5afcc201db737c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b9dc0825e0844f68d5afcc201db737c already in progress: creating tablet
I20251026 02:56:48.422567 3360 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:57038 (request call id 1) took 1994 ms. Trace:
I20251026 02:56:48.422700 3360 rpcz_store.cc:276] 1026 02:56:46.428053 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.428122 (+ 69us) service_pool.cc:225] Handling call
1026 02:56:48.406809 (+1978687us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x616000070954. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 00007fa0a6b2e6bd 00007fa0a6b2e7b3 00007fa0a36f9441 00007fa0ab3af9dd 00007fa0ab544965 00007fa0a2064747 00007fa0a2064475 00007fa09e7cc27e 00007fa09e7cb15e 00007fa09e7d1395 00007fa09e7d1cbf
1026 02:56:48.422397 (+ 15588us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x616000070954. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fa0a36f8945 00007fa0ab32a095 00007fa0ab544cf0 00007fa0a2064747 00007fa0a2064475 00007fa09e7cc27e 00007fa09e7cb15e 00007fa09e7d1395 00007fa09e7d1cbf 00007fa09e7d1ae8
1026 02:56:48.422478 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4583955456}
W20251026 02:56:48.424273 3362 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.2:51761 (request call id 67) took 2287 ms (client timeout 1000 ms). Trace:
W20251026 02:56:48.424409 3362 rpcz_store.cc:269] 1026 02:56:46.136643 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.175637 (+ 38994us) service_pool.cc:225] Handling call
1026 02:56:48.418508 (+2242871us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x616000070954. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 00007fa0a6b2e6bd 00007fa0a6b2e7b3 00007fa0a36f9441 00007fa0ab3af9dd 00007fa0ab536aa8 00007fa0a204fca6 00007fa0a204f9d5 00007fa09e7cc27e 00007fa09e7cb15e 00007fa09e7d1395 00007fa09e7d1cbf
1026 02:56:48.424176 (+ 5668us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5154330368}
W20251026 02:56:48.425837 3354 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.1:42831 (request call id 94) took 1100 ms (client timeout 1000 ms). Trace:
W20251026 02:56:48.425941 3354 rpcz_store.cc:269] 1026 02:56:47.324997 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.325076 (+ 79us) service_pool.cc:225] Handling call
1026 02:56:48.416404 (+1091328us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x616000070954. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 00007fa0a6b2e6bd 00007fa0a6b2e7b3 00007fa0a36f9441 00007fa0ab3af9dd 00007fa0ab536aa8 00007fa0a204fca6 00007fa0a204f9d5 00007fa09e7cc27e 00007fa09e7cb15e 00007fa09e7d1395 00007fa09e7d1cbf
1026 02:56:48.425783 (+ 9379us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2507874048}
W20251026 02:56:48.426815 3353 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.2:51761 (request call id 68) took 1087 ms (client timeout 1000 ms). Trace:
W20251026 02:56:48.426946 3353 rpcz_store.cc:269] 1026 02:56:47.339384 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.339440 (+ 56us) service_pool.cc:225] Handling call
1026 02:56:48.417274 (+1077834us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x616000070954. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 00007fa0a6b2e6bd 00007fa0a6b2e7b3 00007fa0a36f9441 00007fa0ab3af9dd 00007fa0ab536aa8 00007fa0a204fca6 00007fa0a204f9d5 00007fa09e7cc27e 00007fa09e7cb15e 00007fa09e7d1395 00007fa09e7d1cbf
1026 02:56:48.426738 (+ 9464us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2476626816}
W20251026 02:56:48.427660 3356 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.3:53887 (request call id 67) took 1179 ms (client timeout 1000 ms). Trace:
W20251026 02:56:48.427764 3356 rpcz_store.cc:269] 1026 02:56:47.248109 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.248191 (+ 82us) service_pool.cc:225] Handling call
1026 02:56:48.417666 (+1169475us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x616000070954. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 00007fa0a6b2e6bd 00007fa0a6b2e7b3 00007fa0a36f9441 00007fa0ab3af9dd 00007fa0ab536aa8 00007fa0a204fca6 00007fa0a204f9d5 00007fa09e7cc27e 00007fa09e7cb15e 00007fa09e7d1395 00007fa09e7d1cbf
1026 02:56:48.427599 (+ 9933us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2686022016}
I20251026 02:56:48.430733 2378 raft_consensus.cc:359] T 4b5e61afe90e4bc4a2897de21e0b3919 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.430931 2378 raft_consensus.cc:385] T 4b5e61afe90e4bc4a2897de21e0b3919 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.430999 2378 raft_consensus.cc:740] T 4b5e61afe90e4bc4a2897de21e0b3919 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.431675 2375 tablet_bootstrap.cc:492] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:48.431828 2375 ts_tablet_manager.cc:1403] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.164s user 0.003s sys 0.004s
I20251026 02:56:48.432510 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 922) took 4264 ms. Trace:
I20251026 02:56:48.432593 2376 rpcz_store.cc:276] 1026 02:56:44.167711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.167842 (+ 131us) service_pool.cc:225] Handling call
1026 02:56:48.432348 (+4264506us) ts_tablet_manager.cc:1151] Deleting tablet 0c60d8227eed417595ef68d44c68ebcf
1026 02:56:48.432363 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.432489 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4252094}
I20251026 02:56:48.435195 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 923) took 4265 ms. Trace:
I20251026 02:56:48.435914 2376 rpcz_store.cc:276] 1026 02:56:44.170120 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.170178 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:48.434734 (+4264556us) ts_tablet_manager.cc:1151] Deleting tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1
1026 02:56:48.434748 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.435175 (+ 427us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4262375}
I20251026 02:56:48.436666 2378 consensus_queue.cc:260] T 4b5e61afe90e4bc4a2897de21e0b3919 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:48.440059 2195 consensus_peers.cc:597] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99d318aa0d1847c58167bf0af9e75aa2. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:48.441563 2372 raft_consensus.cc:2272] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.442145 2372 ts_tablet_manager.cc:1916] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.444841 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 924) took 4273 ms. Trace:
I20251026 02:56:48.447980 2381 tablet_bootstrap.cc:654] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.448375 2378 ts_tablet_manager.cc:1434] T 4b5e61afe90e4bc4a2897de21e0b3919 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20251026 02:56:48.448604 2375 raft_consensus.cc:359] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.448778 2375 raft_consensus.cc:385] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.448825 2375 raft_consensus.cc:740] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.449020 2375 consensus_queue.cc:260] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.445828 2376 rpcz_store.cc:276] 1026 02:56:44.170882 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.170941 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:48.440730 (+4269789us) ts_tablet_manager.cc:1151] Deleting tablet 63f1abc4477f4198ad2cc8481d69df73
1026 02:56:48.443676 (+ 2946us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.444467 (+ 791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4268178}
I20251026 02:56:48.449532 2375 ts_tablet_manager.cc:1434] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20251026 02:56:48.453846 2377 ts_tablet_manager.cc:1929] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.454039 2377 log.cc:1199] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/c31d2704d1f14ddead69c794cc631ec1
I20251026 02:56:48.454480 2377 ts_tablet_manager.cc:1950] T c31d2704d1f14ddead69c794cc631ec1 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:48.455375 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 947) took 5283 ms. Trace:
I20251026 02:56:48.455526 2377 rpcz_store.cc:276] 1026 02:56:43.171563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.171605 (+ 42us) service_pool.cc:225] Handling call
1026 02:56:48.283843 (+5112238us) ts_tablet_manager.cc:1151] Deleting tablet c31d2704d1f14ddead69c794cc631ec1
1026 02:56:48.283853 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.370376 (+ 86523us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.453731 (+ 83355us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.455352 (+ 1621us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5111947}
I20251026 02:56:48.456106 2377 tablet_replica.cc:333] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:48.458135 2378 tablet_bootstrap.cc:492] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:48.457875 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 925) took 4282 ms. Trace:
I20251026 02:56:48.458876 2376 rpcz_store.cc:276] 1026 02:56:44.174698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.174749 (+ 51us) service_pool.cc:225] Handling call
1026 02:56:48.456551 (+4281802us) ts_tablet_manager.cc:1151] Deleting tablet 5eda85f2d53a43d68907c467c7da9bd6
1026 02:56:48.456561 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.457556 (+ 995us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4281471}
I20251026 02:56:48.461345 3997 raft_consensus.cc:493] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.461520 3997 raft_consensus.cc:515] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.462126 3997 leader_election.cc:290] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.462791 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7834a85168364390a841fc883681c845" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:48.463361 2053 leader_election.cc:343] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:48.433046 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0c60d8227eed417595ef68d44c68ebcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c60d8227eed417595ef68d44c68ebcf already in progress: creating tablet
I20251026 02:56:48.463847 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet c31d2704d1f14ddead69c794cc631ec1 (table default.Table_b6138cb4ba3c4b70bcec42f1ce989668 [id=9b9d43027de84287b1e2fd35f5e4d5f5]) successfully deleted
W20251026 02:56:48.464051 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 already in progress: creating tablet
W20251026 02:56:48.464257 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 63f1abc4477f4198ad2cc8481d69df73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63f1abc4477f4198ad2cc8481d69df73 already in progress: creating tablet
W20251026 02:56:48.464636 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 5eda85f2d53a43d68907c467c7da9bd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5eda85f2d53a43d68907c467c7da9bd6 already in progress: creating tablet
I20251026 02:56:48.466516 2375 tablet_bootstrap.cc:492] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:48.466804 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 926) took 4243 ms. Trace:
I20251026 02:56:48.468544 2375 tablet_bootstrap.cc:654] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.472640 2376 rpcz_store.cc:276] 1026 02:56:44.222965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.223028 (+ 63us) service_pool.cc:225] Handling call
1026 02:56:48.466679 (+4243651us) ts_tablet_manager.cc:1151] Deleting tablet 146d6f13b9ef47f29711a102791bab5a
1026 02:56:48.466691 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.466786 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4230332}
W20251026 02:56:48.473258 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 146d6f13b9ef47f29711a102791bab5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 146d6f13b9ef47f29711a102791bab5a already in progress: creating tablet
I20251026 02:56:48.473702 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 938) took 3729 ms. Trace:
I20251026 02:56:48.473809 2376 rpcz_store.cc:276] 1026 02:56:44.744420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.744498 (+ 78us) service_pool.cc:225] Handling call
1026 02:56:48.473580 (+3729082us) ts_tablet_manager.cc:1151] Deleting tablet e45600a92f8e46b096badd700779856d
1026 02:56:48.473591 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.473686 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3728857}
W20251026 02:56:48.474239 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet e45600a92f8e46b096badd700779856d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e45600a92f8e46b096badd700779856d already in progress: creating tablet
I20251026 02:56:48.474538 2376 tablet_replica.cc:333] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:48.474740 2376 raft_consensus.cc:2243] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.474962 2376 raft_consensus.cc:2272] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.475538 2376 ts_tablet_manager.cc:1916] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.462965 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7834a85168364390a841fc883681c845" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:48.480690 2053 leader_election.cc:343] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.480834 2053 leader_election.cc:304] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.481602 3997 raft_consensus.cc:2749] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.484333 2377 raft_consensus.cc:2243] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:48.484771 2377 raft_consensus.cc:2272] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.485409 2377 ts_tablet_manager.cc:1916] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.469852 2378 tablet_bootstrap.cc:654] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.487434 4040 catalog_manager.cc:1269] Loaded cluster ID: 98a29211b5ce4bf98616dd4b8985c732
I20251026 02:56:48.488268 2381 tablet_bootstrap.cc:492] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:48.488441 2381 ts_tablet_manager.cc:1403] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.002s
I20251026 02:56:48.489218 2381 raft_consensus.cc:359] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.489409 2381 raft_consensus.cc:385] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.489488 2381 raft_consensus.cc:740] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.489673 2381 consensus_queue.cc:260] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.490612 4069 consensus_queue.cc:1048] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:48.490943 2381 ts_tablet_manager.cc:1434] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:48.491245 2381 tablet_bootstrap.cc:492] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:48.497677 2372 ts_tablet_manager.cc:1929] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.497821 2372 log.cc:1199] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/1fb2ff87cf0248a0a4a77f2733ef867b
I20251026 02:56:48.498188 2372 ts_tablet_manager.cc:1950] T 1fb2ff87cf0248a0a4a77f2733ef867b P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:48.501016 4040 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: loading cluster ID for follower catalog manager: success
I20251026 02:56:48.507903 2381 tablet_bootstrap.cc:654] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.500164 2376 ts_tablet_manager.cc:1929] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.509295 2376 log.cc:1199] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/7177931353164b2aa82c00b2807ccd63
I20251026 02:56:48.509697 2376 ts_tablet_manager.cc:1950] T 7177931353164b2aa82c00b2807ccd63 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:48.510032 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 946) took 3499 ms. Trace:
I20251026 02:56:48.510121 2376 rpcz_store.cc:276] 1026 02:56:45.010220 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.010291 (+ 71us) service_pool.cc:225] Handling call
1026 02:56:48.474499 (+3464208us) ts_tablet_manager.cc:1151] Deleting tablet 7177931353164b2aa82c00b2807ccd63
1026 02:56:48.474507 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.483147 (+ 8640us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.500051 (+ 16904us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.510013 (+ 9962us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3463970}
I20251026 02:56:48.510658 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 7177931353164b2aa82c00b2807ccd63 (table default.Table_4bbbdbe2670f4595bf4972ee8b3c3b04 [id=7de20c0869584b56b936144d62202444]) successfully deleted
W20251026 02:56:48.514074 2194 consensus_peers.cc:597] T c0bacd15b41149b5856d20068711b432 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:48.520082 2377 ts_tablet_manager.cc:1929] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.529729 2378 tablet_bootstrap.cc:492] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:48.529966 2378 ts_tablet_manager.cc:1403] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.073s user 0.001s sys 0.006s
I20251026 02:56:48.530804 2378 raft_consensus.cc:359] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.531067 2378 raft_consensus.cc:385] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.531167 2378 raft_consensus.cc:740] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.531599 2378 consensus_queue.cc:260] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.534387 4167 raft_consensus.cc:493] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.534528 4167 raft_consensus.cc:515] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.535099 4167 leader_election.cc:290] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.535782 2378 ts_tablet_manager.cc:1434] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251026 02:56:48.535800 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c7efb70c0804c149f36440a4a9e6ef8" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:48.536029 1916 leader_election.cc:343] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9c7efb70c0804c149f36440a4a9e6ef8
I20251026 02:56:48.536182 2378 tablet_bootstrap.cc:492] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:48.536473 1915 leader_election.cc:304] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:48.536832 4167 raft_consensus.cc:2749] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.538601 2378 tablet_bootstrap.cc:654] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.412940 3391 rpcz_store.cc:276] 1026 02:56:46.121930 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.122356 (+ 426us) service_pool.cc:225] Handling call
1026 02:56:46.140967 (+ 18611us) raft_consensus.cc:1471] Updating replica for 356 ops
1026 02:56:46.165462 (+ 24495us) raft_consensus.cc:1534] Early marking committed up to index 937
1026 02:56:46.165519 (+ 57us) raft_consensus.cc:1539] Triggering prepare for 356 ops
1026 02:56:46.165638 (+ 119us) write_op.cc:270] Start()
1026 02:56:46.165670 (+ 32us) write_op.cc:276] Timestamp: P: 1761447397861056 usec, L: 0
1026 02:56:46.165855 (+ 185us) write_op.cc:270] Start()
1026 02:56:46.165880 (+ 25us) write_op.cc:276] Timestamp: P: 1761447397893635 usec, L: 0
1026 02:56:46.166016 (+ 136us) write_op.cc:270] Start()
1026 02:56:46.166034 (+ 18us) write_op.cc:276] Timestamp: P: 1761447397895846 usec, L: 0
1026 02:56:46.166155 (+ 121us) write_op.cc:270] Start()
1026 02:56:46.166174 (+ 19us) write_op.cc:276] Timestamp: P: 1761447397936952 usec, L: 0
1026 02:56:46.178936 (+ 12762us) write_op.cc:270] Start()
1026 02:56:46.178965 (+ 29us) write_op.cc:276] Timestamp: P: 1761447397960699 usec, L: 0
1026 02:56:46.179134 (+ 169us) write_op.cc:270] Start()
1026 02:56:46.179152 (+ 18us) write_op.cc:276] Timestamp: P: 1761447397961676 usec, L: 0
1026 02:56:46.179276 (+ 124us) write_op.cc:270] Start()
1026 02:56:46.179308 (+ 32us) write_op.cc:276] Timestamp: P: 1761447397975637 usec, L: 0
1026 02:56:46.179466 (+ 158us) write_op.cc:270] Start()
1026 02:56:46.179486 (+ 20us) write_op.cc:276] Timestamp: P: 1761447397976778 usec, L: 0
1026 02:56:46.179632 (+ 146us) write_op.cc:270] Start()
1026 02:56:46.179654 (+ 22us) write_op.cc:276] Timestamp: P: 1761447397995931 usec, L: 0
1026 02:56:46.179779 (+ 125us) write_op.cc:270] Start()
1026 02:56:46.179813 (+ 34us) write_op.cc:276] Timestamp: P: 1761447398014688 usec, L: 0
1026 02:56:46.179956 (+ 143us) write_op.cc:270] Start()
1026 02:56:46.179977 (+ 21us) write_op.cc:276] Timestamp: P: 1761447398037056 usec, L: 0
1026 02:56:46.180096 (+ 119us) write_op.cc:270] Start()
1026 02:56:46.180115 (+ 19us) write_op.cc:276] Timestamp: P: 1761447398061618 usec, L: 0
1026 02:56:46.180281 (+ 166us) write_op.cc:270] Start()
1026 02:56:46.180302 (+ 21us) write_op.cc:276] Timestamp: P: 1761447398076299 usec, L: 0
1026 02:56:46.180438 (+ 136us) write_op.cc:270] Start()
1026 02:56:46.180457 (+ 19us) write_op.cc:276] Timestamp: P: 1761447398092030 usec, L: 0
1026 02:56:46.180576 (+ 119us) write_op.cc:270] Start()
1026 02:56:46.180593 (+ 17us) write_op.cc:276] Timestamp: P: 1761447398094842 usec, L: 0
1026 02:56:46.180726 (+ 133us) write_op.cc:270] Start()
1026 02:56:46.180747 (+ 21us) write_op.cc:276] Timestamp: P: 1761447398113665 usec, L: 0
1026 02:56:46.180882 (+ 135us) write_op.cc:270] Start()
1026 02:56:46.180904 (+ 22us) write_op.cc:276] Timestamp: P: 1761447398138285 usec, L: 0
1026 02:56:46.181059 (+ 155us) write_op.cc:270] Start()
1026 02:56:46.181081 (+ 22us) write_op.cc:276] Timestamp: P: 1761447398158249 usec, L: 0
1026 02:56:46.181233 (+ 152us) write_op.cc:270] Start()
1026 02:56:46.181254 (+ 21us) write_op.cc:276] Timestamp: P: 1761447398165076 usec, L: 0
1026 02:56:46.181378 (+ 124us) write_op.cc:270] Start()
1026 02:56:46.181396 (+ 18us) write_op.cc:276] Timestamp: P: 1761447398166414 usec, L: 0
1026 02:56:46.181600 (+ 204us) write_op.cc:270] Start()
1026 02:56:46.181619 (+ 19us) write_op.cc:276] Timestamp: P: 1761447398167375 usec, L: 0
1026 02:56:46.181742 (+ 123us) write_op.cc:270] Start()
1026 02:56:46.181760 (+ 18us) write_op.cc:276] Timestamp: P: 1761447398168324 usec, L: 0
1026 02:56:46.181891 (+ 131us) write_op.cc:270] Start()
1026 02:56:46.181910 (+ 19us) write_op.cc:276] Timestamp: P: 1761447398169315 usec, L: 0
1026 02:56:46.182098 (+ 188us) write_op.cc:270] Start()
1026 02:56:46.182118 (+ 20us) write_op.cc:276] Timestamp: P: 1761447398291129 usec, L: 0
1026 02:56:46.182281 (+ 163us) write_op.cc:270] Start()
1026 02:56:46.182304 (+ 23us) write_op.cc:276] Timestamp: P: 1761447398291791 usec, L: 0
1026 02:56:46.190821 (+ 8517us) write_op.cc:270] Start()
1026 02:56:46.190854 (+ 33us) write_op.cc:276] Timestamp: P: 1761447398292372 usec, L: 0
1026 02:56:46.191046 (+ 192us) write_op.cc:270] Start()
1026 02:56:46.191068 (+ 22us) write_op.cc:276] Timestamp: P: 1761447398292892 usec, L: 0
1026 02:56:46.191206 (+ 138us) write_op.cc:270] Start()
1026 02:56:46.191224 (+ 18us) write_op.cc:276] Timestamp: P: 1761447398296145 usec, L: 0
1026 02:56:46.191348 (+ 124us) write_op.cc:270] Start()
1026 02:56:46.191366 (+ 18us) write_op.cc:276] Timestamp: P: 1761447398298957 usec, L: 0
1026 02:56:46.191527 (+ 161us) write_op.cc:270] Start()
1026 02:56:46.191548 (+ 21us) write_op.cc:276] Timestamp: P: 1761447398335978 usec, L: 0
1026 02:56:46.191715 (+ 167us) write_op.cc:270] Start()
1026 02:56:46.191738 (+ 23us) write_op.cc:276] Timestamp: P: 1761447398428096 usec, L: 0
1026 02:56:46.191905 (+ 167us) write_op.cc:270] Start()
1026 02:56:46.191925 (+ 20us) write_op.cc:276] Timestamp: P: 1761447398434596 usec, L: 0
1026 02:56:46.192069 (+ 144us) write_op.cc:270] Start()
1026 02:56:46.192105 (+ 36us) write_op.cc:276] Timestamp: P: 1761447398435218 usec, L: 0
1026 02:56:46.192296 (+ 191us) write_op.cc:270] Start()
1026 02:56:46.192315 (+ 19us) write_op.cc:276] Timestamp: P: 1761447398477734 usec, L: 0
1026 02:56:46.192440 (+ 125us) write_op.cc:270] Start()
1026 02:56:46.192456 (+ 16us) write_op.cc:276] Timestamp: P: 1761447398478458 usec, L: 0
1026 02:56:46.192574 (+ 118us) write_op.cc:270] Start()
1026 02:56:46.192590 (+ 16us) write_op.cc:276] Timestamp: P: 1761447398479017 usec, L: 0
1026 02:56:46.192703 (+ 113us) write_op.cc:270] Start()
1026 02:56:46.192718 (+ 15us) write_op.cc:276] Timestamp: P: 1761447398479906 usec, L: 0
1026 02:56:46.192822 (+ 104us) write_op.cc:270] Start()
1026 02:56:46.192838 (+ 16us) write_op.cc:276] Timestamp: P: 1761447398518439 usec, L: 0
1026 02:56:46.192983 (+ 145us) write_op.cc:270] Start()
1026 02:56:46.192999 (+ 16us) write_op.cc:276] Timestamp: P: 1761447398532243 usec, L: 0
1026 02:56:46.193118 (+ 119us) write_op.cc:270] Start()
1026 02:56:46.193133 (+ 15us) write_op.cc:276] Timestamp: P: 1761447398542136 usec, L: 0
1026 02:56:46.193240 (+ 107us) write_op.cc:270] Start()
1026 02:56:46.193256 (+ 16us) write_op.cc:276] Timestamp: P: 1761447398559432 usec, L: 0
1026 02:56:46.193356 (+ 100us) write_op.cc:270] Start()
1026 02:56:46.193373 (+ 17us) write_op.cc:276] Timestamp: P: 1761447398580677 usec, L: 0
1026 02:56:46.193514 (+ 141us) write_op.cc:270] Start()
1026 02:56:46.193534 (+ 20us) write_op.cc:276] Timestamp: P: 1761447398609827 usec, L: 0
1026 02:56:46.193642 (+ 108us) write_op.cc:270] Start()
1026 02:56:46.193657 (+ 15us) write_op.cc:276] Timestamp: P: 1761447398619397 usec, L: 0
1026 02:56:46.193764 (+ 107us) write_op.cc:270] Start()
1026 02:56:46.193781 (+ 17us) write_op.cc:276] Timestamp: P: 1761447398621755 usec, L: 0
1026 02:56:46.193890 (+ 109us) write_op.cc:270] Start()
1026 02:56:46.193904 (+ 14us) write_op.cc:276] Timestamp: P: 1761447398625492 usec, L: 0
1026 02:56:46.193999 (+ 95us) write_op.cc:270] Start()
1026 02:56:46.194014 (+ 15us) write_op.cc:276] Timestamp: P: 1761447398626039 usec, L: 0
1026 02:56:46.194140 (+ 126us) write_op.cc:270] Start()
1026 02:56:46.194156 (+ 16us) write_op.cc:276] Timestamp: P: 1761447398632208 usec, L: 0
1026 02:56:46.194283 (+ 127us) write_op.cc:270] Start()
1026 02:56:46.194305 (+ 22us) write_op.cc:276] Timestamp: P: 1761447398632839 usec, L: 0
1026 02:56:46.194450 (+ 145us) write_op.cc:270] Start()
1026 02:56:46.194471 (+ 21us) write_op.cc:276] Timestamp: P: 1761447398633425 usec, L: 0
1026 02:56:46.194602 (+ 131us) write_op.cc:270] Start()
1026 02:56:46.194619 (+ 17us) write_op.cc:276] Timestamp: P: 1761447398645550 usec, L: 0
1026 02:56:46.194748 (+ 129us) write_op.cc:270] Start()
1026 02:56:46.194770 (+ 22us) write_op.cc:276] Timestamp: P: 1761447398659850 usec, L: 0
1026 02:56:46.194913 (+ 143us) write_op.cc:270] Start()
1026 02:56:46.194931 (+ 18us) write_op.cc:276] Timestamp: P: 1761447398665878 usec, L: 0
1026 02:56:46.195032 (+ 101us) write_op.cc:270] Start()
1026 02:56:46.195053 (+ 21us) write_op.cc:276] Timestamp: P: 1761447398683406 usec, L: 0
1026 02:56:46.195172 (+ 119us) write_op.cc:270] Start()
1026 02:56:46.195206 (+ 34us) write_op.cc:276] Timestamp: P: 1761447398684470 usec, L: 0
1026 02:56:46.195350 (+ 144us) write_op.cc:270] Start()
1026 02:56:46.195368 (+ 18us) write_op.cc:276] Timestamp: P: 1761447398686168 usec, L: 0
1026 02:56:46.195506 (+ 138us) write_op.cc:270] Start()
1026 02:56:46.195528 (+ 22us) write_op.cc:276] Timestamp: P: 1761447398696837 usec, L: 0
1026 02:56:46.195657 (+ 129us) write_op.cc:270] Start()
1026 02:56:46.195676 (+ 19us) write_op.cc:276] Timestamp: P: 1761447398697984 usec, L: 0
1026 02:56:46.195825 (+ 149us) write_op.cc:270] Start()
1026 02:56:46.195843 (+ 18us) write_op.cc:276] Timestamp: P: 1761447398712921 usec, L: 0
1026 02:56:46.195970 (+ 127us) write_op.cc:270] Start()
1026 02:56:46.196015 (+ 45us) write_op.cc:276] Timestamp: P: 1761447398720874 usec, L: 0
1026 02:56:46.196166 (+ 151us) write_op.cc:270] Start()
1026 02:56:46.196187 (+ 21us) write_op.cc:276] Timestamp: P: 1761447398742814 usec, L: 0
1026 02:56:46.262636 (+ 66449us) write_op.cc:270] Start()
1026 02:56:46.262665 (+ 29us) write_op.cc:276] Timestamp: P: 1761447398755586 usec, L: 0
1026 02:56:46.262800 (+ 135us) write_op.cc:270] Start()
1026 02:56:46.262817 (+ 17us) write_op.cc:276] Timestamp: P: 1761447398757571 usec, L: 0
1026 02:56:46.262928 (+ 111us) write_op.cc:270] Start()
1026 02:56:46.262944 (+ 16us) write_op.cc:276] Timestamp: P: 1761447398758217 usec, L: 0
1026 02:56:46.263105 (+ 161us) write_op.cc:270] Start()
1026 02:56:46.263123 (+ 18us) write_op.cc:276] Timestamp: P: 1761447398758720 usec, L: 0
1026 02:56:46.263270 (+ 147us) write_op.cc:270] Start()
1026 02:56:46.263294 (+ 24us) write_op.cc:276] Timestamp: P: 1761447398801910 usec, L: 0
1026 02:56:46.263442 (+ 148us) write_op.cc:270] Start()
1026 02:56:46.263467 (+ 25us) write_op.cc:276] Timestamp: P: 1761447398803862 usec, L: 0
1026 02:56:46.263613 (+ 146us) write_op.cc:270] Start()
1026 02:56:46.263633 (+ 20us) write_op.cc:276] Timestamp: P: 1761447398853896 usec, L: 0
1026 02:56:46.263773 (+ 140us) write_op.cc:270] Start()
1026 02:56:46.263795 (+ 22us) write_op.cc:276] Timestamp: P: 1761447398855139 usec, L: 0
1026 02:56:46.263938 (+ 143us) write_op.cc:270] Start()
1026 02:56:46.263955 (+ 17us) write_op.cc:276] Timestamp: P: 1761447398855712 usec, L: 0
1026 02:56:46.264093 (+ 138us) write_op.cc:270] Start()
1026 02:56:46.264112 (+ 19us) write_op.cc:276] Timestamp: P: 1761447398856282 usec, L: 0
1026 02:56:46.264264 (+ 152us) write_op.cc:270] Start()
1026 02:56:46.264283 (+ 19us) write_op.cc:276] Timestamp: P: 1761447398863332 usec, L: 0
1026 02:56:46.264406 (+ 123us) write_op.cc:270] Start()
1026 02:56:46.264452 (+ 46us) write_op.cc:276] Timestamp: P: 1761447399100263 usec, L: 0
1026 02:56:46.264582 (+ 130us) write_op.cc:270] Start()
1026 02:56:46.264601 (+ 19us) write_op.cc:276] Timestamp: P: 1761447399101322 usec, L: 0
1026 02:56:46.264714 (+ 113us) write_op.cc:270] Start()
1026 02:56:46.264731 (+ 17us) write_op.cc:276] Timestamp: P: 1761447399101841 usec, L: 0
1026 02:56:46.264861 (+ 130us) write_op.cc:270] Start()
1026 02:56:46.264883 (+ 22us) write_op.cc:276] Timestamp: P: 1761447399104081 usec, L: 0
1026 02:56:46.265032 (+ 149us) write_op.cc:270] Start()
1026 02:56:46.265051 (+ 19us) write_op.cc:276] Timestamp: P: 1761447399106492 usec, L: 0
1026 02:56:46.265152 (+ 101us) write_op.cc:270] Start()
1026 02:56:46.265167 (+ 15us) write_op.cc:276] Timestamp: P: 1761447399119248 usec, L: 0
1026 02:56:46.265266 (+ 99us) write_op.cc:270] Start()
1026 02:56:46.265287 (+ 21us) write_op.cc:276] Timestamp: P: 1761447399119875 usec, L: 0
1026 02:56:46.265389 (+ 102us) write_op.cc:270] Start()
1026 02:56:46.265406 (+ 17us) write_op.cc:276] Timestamp: P: 1761447399121166 usec, L: 0
1026 02:56:46.265522 (+ 116us) write_op.cc:270] Start()
1026 02:56:46.265551 (+ 29us) write_op.cc:276] Timestamp: P: 1761447399121846 usec, L: 0
1026 02:56:46.265695 (+ 144us) write_op.cc:270] Start()
1026 02:56:46.265713 (+ 18us) write_op.cc:276] Timestamp: P: 1761447399122379 usec, L: 0
1026 02:56:46.265828 (+ 115us) write_op.cc:270] Start()
1026 02:56:46.265845 (+ 17us) write_op.cc:276] Timestamp: P: 1761447399138841 usec, L: 0
1026 02:56:46.265960 (+ 115us) write_op.cc:270] Start()
1026 02:56:46.265977 (+ 17us) write_op.cc:276] Timestamp: P: 1761447399146280 usec, L: 0
1026 02:56:46.266120 (+ 143us) write_op.cc:270] Start()
1026 02:56:46.266139 (+ 19us) write_op.cc:276] Timestamp: P: 1761447399153615 usec, L: 0
1026 02:56:46.266265 (+ 126us) write_op.cc:270] Start()
1026 02:56:46.266284 (+ 19us) write_op.cc:276] Timestamp: P: 1761447399160558 usec, L: 0
1026 02:56:46.266402 (+ 118us) write_op.cc:270] Start()
1026 02:56:46.266428 (+ 26us) write_op.cc:276] Timestamp: P: 1761447399172572 usec, L: 0
1026 02:56:46.266568 (+ 140us) write_op.cc:270] Start()
1026 02:56:46.266586 (+ 18us) write_op.cc:276] Timestamp: P: 1761447399194715 usec, L: 0
1026 02:56:46.266702 (+ 116us) write_op.cc:270] Start()
1026 02:56:46.266718 (+ 16us) write_op.cc:276] Timestamp: P: 1761447399210145 usec, L: 0
1026 02:56:46.266872 (+ 154us) write_op.cc:270] Start()
1026 02:56:46.266891 (+ 19us) write_op.cc:276] Timestamp: P: 1761447399365514 usec, L: 0
1026 02:56:46.267010 (+ 119us) write_op.cc:270] Start()
1026 02:56:46.267028 (+ 18us) write_op.cc:276] Timestamp: P: 1761447399528180 usec, L: 0
1026 02:56:46.267138 (+ 110us) write_op.cc:270] Start()
1026 02:56:46.267156 (+ 18us) write_op.cc:276] Timestamp: P: 1761447399541800 usec, L: 0
1026 02:56:46.267282 (+ 126us) write_op.cc:270] Start()
1026 02:56:46.267299 (+ 17us) write_op.cc:276] Timestamp: P: 1761447399591047 usec, L: 0
1026 02:56:46.267463 (+ 164us) write_op.cc:270] Start()
1026 02:56:46.267487 (+ 24us) write_op.cc:276] Timestamp: P: 1761447399607580 usec, L: 0
1026 02:56:46.267631 (+ 144us) write_op.cc:270] Start()
1026 02:56:46.267652 (+ 21us) write_op.cc:276] Timestamp: P: 1761447399640234 usec, L: 0
1026 02:56:46.267822 (+ 170us) write_op.cc:270] Start()
1026 02:56:46.267846 (+ 24us) write_op.cc:276] Timestamp: P: 1761447399641691 usec, L: 0
1026 02:56:46.267988 (+ 142us) write_op.cc:270] Start()
1026 02:56:46.268008 (+ 20us) write_op.cc:276] Timestamp: P: 1761447399691564 usec, L: 0
1026 02:56:46.488588 (+220580us) write_op.cc:270] Start()
1026 02:56:46.488625 (+ 37us) write_op.cc:276] Timestamp: P: 1761447399708425 usec, L: 0
1026 02:56:46.488846 (+ 221us) write_op.cc:270] Start()
1026 02:56:46.488867 (+ 21us) write_op.cc:276] Timestamp: P: 1761447399722416 usec, L: 0
1026 02:56:46.489468 (+ 601us) write_op.cc:270] Start()
1026 02:56:46.489497 (+ 29us) write_op.cc:276] Timestamp: P: 1761447399898879 usec, L: 0
1026 02:56:46.489799 (+ 302us) write_op.cc:270] Start()
1026 02:56:46.489823 (+ 24us) write_op.cc:276] Timestamp: P: 1761447399928914 usec, L: 0
1026 02:56:46.489971 (+ 148us) write_op.cc:270] Start()
1026 02:56:46.490003 (+ 32us) write_op.cc:276] Timestamp: P: 1761447399930316 usec, L: 0
1026 02:56:46.490144 (+ 141us) write_op.cc:270] Start()
1026 02:56:46.490161 (+ 17us) write_op.cc:276] Timestamp: P: 1761447399931308 usec, L: 0
1026 02:56:46.490271 (+ 110us) write_op.cc:270] Start()
1026 02:56:46.490287 (+ 16us) write_op.cc:276] Timestamp: P: 1761447399932127 usec, L: 0
1026 02:56:46.490392 (+ 105us) write_op.cc:270] Start()
1026 02:56:46.490406 (+ 14us) write_op.cc:276] Timestamp: P: 1761447399937159 usec, L: 0
1026 02:56:46.490513 (+ 107us) write_op.cc:270] Start()
1026 02:56:46.490529 (+ 16us) write_op.cc:276] Timestamp: P: 1761447399958218 usec, L: 0
1026 02:56:46.490637 (+ 108us) write_op.cc:270] Start()
1026 02:56:46.490653 (+ 16us) write_op.cc:276] Timestamp: P: 1761447399999070 usec, L: 0
1026 02:56:46.490754 (+ 101us) write_op.cc:270] Start()
1026 02:56:46.490769 (+ 15us) write_op.cc:276] Timestamp: P: 1761447400075597 usec, L: 0
1026 02:56:46.490876 (+ 107us) write_op.cc:270] Start()
1026 02:56:46.490892 (+ 16us) write_op.cc:276] Timestamp: P: 1761447400078939 usec, L: 0
1026 02:56:46.491017 (+ 125us) write_op.cc:270] Start()
1026 02:56:46.491033 (+ 16us) write_op.cc:276] Timestamp: P: 1761447400118870 usec, L: 0
1026 02:56:46.491134 (+ 101us) write_op.cc:270] Start()
1026 02:56:46.491148 (+ 14us) write_op.cc:276] Timestamp: P: 1761447400123856 usec, L: 0
1026 02:56:46.491286 (+ 138us) write_op.cc:270] Start()
1026 02:56:46.491305 (+ 19us) write_op.cc:276] Timestamp: P: 1761447400124793 usec, L: 0
1026 02:56:46.491450 (+ 145us) write_op.cc:270] Start()
1026 02:56:46.491470 (+ 20us) write_op.cc:276] Timestamp: P: 1761447400125582 usec, L: 0
1026 02:56:46.491581 (+ 111us) write_op.cc:270] Start()
1026 02:56:46.491597 (+ 16us) write_op.cc:276] Timestamp: P: 1761447400126868 usec, L: 0
1026 02:56:46.491717 (+ 120us) write_op.cc:270] Start()
1026 02:56:46.491733 (+ 16us) write_op.cc:276] Timestamp: P: 1761447400306289 usec, L: 0
1026 02:56:46.491830 (+ 97us) write_op.cc:270] Start()
1026 02:56:46.491845 (+ 15us) write_op.cc:276] Timestamp: P: 1761447400310722 usec, L: 0
1026 02:56:46.491948 (+ 103us) write_op.cc:270] Start()
1026 02:56:46.491964 (+ 16us) write_op.cc:276] Timestamp: P: 1761447400311308 usec, L: 0
1026 02:56:46.492073 (+ 109us) write_op.cc:270] Start()
1026 02:56:46.492088 (+ 15us) write_op.cc:276] Timestamp: P: 1761447400311884 usec, L: 0
1026 02:56:46.492178 (+ 90us) write_op.cc:270] Start()
1026 02:56:46.492194 (+ 16us) write_op.cc:276] Timestamp: P: 1761447400312461 usec, L: 0
1026 02:56:46.625182 (+132988us) write_op.cc:270] Start()
1026 02:56:46.646705 (+ 21523us) write_op.cc:276] Timestamp: P: 1761447400312991 usec, L: 0
1026 02:56:46.647290 (+ 585us) write_op.cc:270] Start()
1026 02:56:46.647364 (+ 74us) write_op.cc:276] Timestamp: P: 1761447400313892 usec, L: 0
1026 02:56:46.648063 (+ 699us) write_op.cc:270] Start()
1026 02:56:46.648144 (+ 81us) write_op.cc:276] Timestamp: P: 1761447400314386 usec, L: 0
1026 02:56:46.648768 (+ 624us) write_op.cc:270] Start()
1026 02:56:46.648846 (+ 78us) write_op.cc:276] Timestamp: P: 1761447400382828 usec, L: 0
1026 02:56:46.649407 (+ 561us) write_op.cc:270] Start()
1026 02:56:46.649475 (+ 68us) write_op.cc:276] Timestamp: P: 1761447400407698 usec, L: 0
1026 02:56:46.649909 (+ 434us) write_op.cc:270] Start()
1026 02:56:46.649975 (+ 66us) write_op.cc:276] Timestamp: P: 1761447400432933 usec, L: 0
1026 02:56:46.650332 (+ 357us) write_op.cc:270] Start()
1026 02:56:46.650397 (+ 65us) write_op.cc:276] Timestamp: P: 1761447400462257 usec, L: 0
1026 02:56:46.650845 (+ 448us) write_op.cc:270] Start()
1026 02:56:46.650916 (+ 71us) write_op.cc:276] Timestamp: P: 1761447400471349 usec, L: 0
1026 02:56:46.651373 (+ 457us) write_op.cc:270] Start()
1026 02:56:46.651568 (+ 195us) write_op.cc:276] Timestamp: P: 1761447400479619 usec, L: 0
1026 02:56:46.734701 (+ 83133us) write_op.cc:270] Start()
1026 02:56:46.734751 (+ 50us) write_op.cc:276] Timestamp: P: 1761447400490144 usec, L: 0
1026 02:56:46.734994 (+ 243us) write_op.cc:270] Start()
1026 02:56:46.735013 (+ 19us) write_op.cc:276] Timestamp: P: 1761447400493436 usec, L: 0
1026 02:56:46.735129 (+ 116us) write_op.cc:270] Start()
1026 02:56:46.735173 (+ 44us) write_op.cc:276] Timestamp: P: 1761447400504892 usec, L: 0
1026 02:56:46.735304 (+ 131us) write_op.cc:270] Start()
1026 02:56:46.735320 (+ 16us) write_op.cc:276] Timestamp: P: 1761447400525208 usec, L: 0
1026 02:56:46.735422 (+ 102us) write_op.cc:270] Start()
1026 02:56:46.735437 (+ 15us) write_op.cc:276] Timestamp: P: 1761447400534434 usec, L: 0
1026 02:56:46.735575 (+ 138us) write_op.cc:270] Start()
1026 02:56:46.735600 (+ 25us) write_op.cc:276] Timestamp: P: 1761447400546004 usec, L: 0
1026 02:56:46.735717 (+ 117us) write_op.cc:270] Start()
1026 02:56:46.735741 (+ 24us) write_op.cc:276] Timestamp: P: 1761447400569254 usec, L: 0
1026 02:56:46.735929 (+ 188us) write_op.cc:270] Start()
1026 02:56:46.735958 (+ 29us) write_op.cc:276] Timestamp: P: 1761447400621171 usec, L: 0
1026 02:56:46.736103 (+ 145us) write_op.cc:270] Start()
1026 02:56:46.736126 (+ 23us) write_op.cc:276] Timestamp: P: 1761447400637968 usec, L: 0
1026 02:56:46.736301 (+ 175us) write_op.cc:270] Start()
1026 02:56:46.736326 (+ 25us) write_op.cc:276] Timestamp: P: 1761447400657689 usec, L: 0
1026 02:56:46.736501 (+ 175us) write_op.cc:270] Start()
1026 02:56:46.736524 (+ 23us) write_op.cc:276] Timestamp: P: 1761447400658443 usec, L: 0
1026 02:56:46.736671 (+ 147us) write_op.cc:270] Start()
1026 02:56:46.736704 (+ 33us) write_op.cc:276] Timestamp: P: 1761447400659043 usec, L: 0
1026 02:56:46.736992 (+ 288us) write_op.cc:270] Start()
1026 02:56:46.737016 (+ 24us) write_op.cc:276] Timestamp: P: 1761447400710193 usec, L: 0
1026 02:56:46.737131 (+ 115us) write_op.cc:270] Start()
1026 02:56:46.737147 (+ 16us) write_op.cc:276] Timestamp: P: 1761447400721792 usec, L: 0
1026 02:56:46.737241 (+ 94us) write_op.cc:270] Start()
1026 02:56:46.737256 (+ 15us) write_op.cc:276] Timestamp: P: 1761447400750955 usec, L: 0
1026 02:56:46.737368 (+ 112us) write_op.cc:270] Start()
1026 02:56:46.737383 (+ 15us) write_op.cc:276] Timestamp: P: 1761447400753230 usec, L: 0
1026 02:56:46.737539 (+ 156us) write_op.cc:270] Start()
1026 02:56:46.737559 (+ 20us) write_op.cc:276] Timestamp: P: 1761447400753892 usec, L: 0
1026 02:56:46.737662 (+ 103us) write_op.cc:270] Start()
1026 02:56:46.737702 (+ 40us) write_op.cc:276] Timestamp: P: 1761447400754436 usec, L: 0
1026 02:56:46.737807 (+ 105us) write_op.cc:270] Start()
1026 02:56:46.737822 (+ 15us) write_op.cc:276] Timestamp: P: 1761447400808103 usec, L: 0
1026 02:56:46.737920 (+ 98us) write_op.cc:270] Start()
1026 02:56:46.737935 (+ 15us) write_op.cc:276] Timestamp: P: 1761447400812823 usec, L: 0
1026 02:56:46.738056 (+ 121us) write_op.cc:270] Start()
1026 02:56:46.738072 (+ 16us) write_op.cc:276] Timestamp: P: 1761447400815515 usec, L: 0
1026 02:56:46.738178 (+ 106us) write_op.cc:270] Start()
1026 02:56:46.738193 (+ 15us) write_op.cc:276] Timestamp: P: 1761447400845170 usec, L: 0
1026 02:56:46.738303 (+ 110us) write_op.cc:270] Start()
1026 02:56:46.738321 (+ 18us) write_op.cc:276] Timestamp: P: 1761447400859192 usec, L: 0
1026 02:56:46.738432 (+ 111us) write_op.cc:270] Start()
1026 02:56:46.738449 (+ 17us) write_op.cc:276] Timestamp: P: 1761447400871846 usec, L: 0
1026 02:56:46.738569 (+ 120us) write_op.cc:270] Start()
1026 02:56:46.738586 (+ 17us) write_op.cc:276] Timestamp: P: 1761447400882356 usec, L: 0
1026 02:56:46.738748 (+ 162us) write_op.cc:270] Start()
1026 02:56:46.738772 (+ 24us) write_op.cc:276] Timestamp: P: 1761447400889735 usec, L: 0
1026 02:56:46.738903 (+ 131us) write_op.cc:270] Start()
1026 02:56:46.738926 (+ 23us) write_op.cc:276] Timestamp: P: 1761447400894890 usec, L: 0
1026 02:56:46.739093 (+ 167us) write_op.cc:270] Start()
1026 02:56:46.739119 (+ 26us) write_op.cc:276] Timestamp: P: 1761447400899009 usec, L: 0
1026 02:56:46.739279 (+ 160us) write_op.cc:270] Start()
1026 02:56:46.739302 (+ 23us) write_op.cc:276] Timestamp: P: 1761447400902492 usec, L: 0
1026 02:56:46.739452 (+ 150us) write_op.cc:270] Start()
1026 02:56:46.739492 (+ 40us) write_op.cc:276] Timestamp: P: 1761447400961624 usec, L: 0
1026 02:56:46.739639 (+ 147us) write_op.cc:270] Start()
1026 02:56:46.739654 (+ 15us) write_op.cc:276] Timestamp: P: 1761447400974333 usec, L: 0
1026 02:56:46.739803 (+ 149us) write_op.cc:270] Start()
1026 02:56:46.739826 (+ 23us) write_op.cc:276] Timestamp: P: 1761447400987533 usec, L: 0
1026 02:56:46.739998 (+ 172us) write_op.cc:270] Start()
1026 02:56:46.740016 (+ 18us) write_op.cc:276] Timestamp: P: 1761447401003050 usec, L: 0
1026 02:56:46.740129 (+ 113us) write_op.cc:270] Start()
1026 02:56:46.740147 (+ 18us) write_op.cc:276] Timestamp: P: 1761447401019859 usec, L: 0
1026 02:56:46.907575 (+167428us) write_op.cc:270] Start()
1026 02:56:46.907625 (+ 50us) write_op.cc:276] Timestamp: P: 1761447401041543 usec, L: 0
1026 02:56:46.907905 (+ 280us) write_op.cc:270] Start()
1026 02:56:46.907938 (+ 33us) write_op.cc:276] Timestamp: P: 1761447401050900 usec, L: 0
1026 02:56:46.908114 (+ 176us) write_op.cc:270] Start()
1026 02:56:46.908140 (+ 26us) write_op.cc:276] Timestamp: P: 1761447401107627 usec, L: 0
1026 02:56:46.908319 (+ 179us) write_op.cc:270] Start()
1026 02:56:46.908348 (+ 29us) write_op.cc:276] Timestamp: P: 1761447401111311 usec, L: 0
1026 02:56:46.908560 (+ 212us) write_op.cc:270] Start()
1026 02:56:46.908585 (+ 25us) write_op.cc:276] Timestamp: P: 1761447401114629 usec, L: 0
1026 02:56:46.908752 (+ 167us) write_op.cc:270] Start()
1026 02:56:46.908775 (+ 23us) write_op.cc:276] Timestamp: P: 1761447401117767 usec, L: 0
1026 02:56:46.909289 (+ 514us) write_op.cc:270] Start()
1026 02:56:46.909642 (+ 353us) write_op.cc:276] Timestamp: P: 1761447401122955 usec, L: 0
1026 02:56:46.910424 (+ 782us) write_op.cc:270] Start()
1026 02:56:46.910714 (+ 290us) write_op.cc:276] Timestamp: P: 1761447401123683 usec, L: 0
1026 02:56:46.911538 (+ 824us) write_op.cc:270] Start()
1026 02:56:46.911619 (+ 81us) write_op.cc:276] Timestamp: P: 1761447401162750 usec, L: 0
1026 02:56:46.956081 (+ 44462us) write_op.cc:270] Start()
1026 02:56:46.956539 (+ 458us) write_op.cc:276] Timestamp: P: 1761447401248298 usec, L: 0
1026 02:56:46.983603 (+ 27064us) write_op.cc:270] Start()
1026 02:56:46.984019 (+ 416us) write_op.cc:276] Timestamp: P: 1761447401249220 usec, L: 0
1026 02:56:47.012661 (+ 28642us) write_op.cc:270] Start()
1026 02:56:47.013071 (+ 410us) write_op.cc:276] Timestamp: P: 1761447401249909 usec, L: 0
1026 02:56:47.017676 (+ 4605us) write_op.cc:270] Start()
1026 02:56:47.018109 (+ 433us) write_op.cc:276] Timestamp: P: 1761447401250628 usec, L: 0
1026 02:56:47.093584 (+ 75475us) write_op.cc:270] Start()
1026 02:56:47.094985 (+ 1401us) write_op.cc:276] Timestamp: P: 1761447401479609 usec, L: 0
1026 02:56:47.112107 (+ 17122us) write_op.cc:270] Start()
1026 02:56:47.117201 (+ 5094us) write_op.cc:276] Timestamp: P: 1761447401515711 usec, L: 0
1026 02:56:47.123006 (+ 5805us) write_op.cc:270] Start()
1026 02:56:47.123212 (+ 206us) write_op.cc:276] Timestamp: P: 1761447401517694 usec, L: 0
1026 02:56:47.154423 (+ 31211us) write_op.cc:270] Start()
1026 02:56:47.154466 (+ 43us) write_op.cc:276] Timestamp: P: 1761447401519757 usec, L: 0
1026 02:56:47.154670 (+ 204us) write_op.cc:270] Start()
1026 02:56:47.154689 (+ 19us) write_op.cc:276] Timestamp: P: 1761447401545966 usec, L: 0
1026 02:56:47.154807 (+ 118us) write_op.cc:270] Start()
1026 02:56:47.154824 (+ 17us) write_op.cc:276] Timestamp: P: 1761447401547442 usec, L: 0
1026 02:56:47.154943 (+ 119us) write_op.cc:270] Start()
1026 02:56:47.154960 (+ 17us) write_op.cc:276] Timestamp: P: 1761447401581890 usec, L: 0
1026 02:56:47.155071 (+ 111us) write_op.cc:270] Start()
1026 02:56:47.155087 (+ 16us) write_op.cc:276] Timestamp: P: 1761447401586099 usec, L: 0
1026 02:56:47.155204 (+ 117us) write_op.cc:270] Start()
1026 02:56:47.155221 (+ 17us) write_op.cc:276] Timestamp: P: 1761447401600616 usec, L: 0
1026 02:56:47.155362 (+ 141us) write_op.cc:270] Start()
1026 02:56:47.155384 (+ 22us) write_op.cc:276] Timestamp: P: 1761447401616319 usec, L: 0
1026 02:56:47.155546 (+ 162us) write_op.cc:270] Start()
1026 02:56:47.155569 (+ 23us) write_op.cc:276] Timestamp: P: 1761447401616973 usec, L: 0
1026 02:56:47.155693 (+ 124us) write_op.cc:270] Start()
1026 02:56:47.155713 (+ 20us) write_op.cc:276] Timestamp: P: 1761447401617530 usec, L: 0
1026 02:56:47.155847 (+ 134us) write_op.cc:270] Start()
1026 02:56:47.155870 (+ 23us) write_op.cc:276] Timestamp: P: 1761447401620727 usec, L: 0
1026 02:56:47.156012 (+ 142us) write_op.cc:270] Start()
1026 02:56:47.156034 (+ 22us) write_op.cc:276] Timestamp: P: 1761447401624805 usec, L: 0
1026 02:56:47.156156 (+ 122us) write_op.cc:270] Start()
1026 02:56:47.156180 (+ 24us) write_op.cc:276] Timestamp: P: 1761447401625340 usec, L: 0
1026 02:56:47.156347 (+ 167us) write_op.cc:270] Start()
1026 02:56:47.156365 (+ 18us) write_op.cc:276] Timestamp: P: 1761447401625963 usec, L: 0
1026 02:56:47.156504 (+ 139us) write_op.cc:270] Start()
1026 02:56:47.187656 (+ 31152us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x619000110058. stack: 00007fa0978309f6 00007fa0978307b0 00007fa09680c3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fa0a4be5a1a 00007fa0a4becaff 00007fa0a496616c 00007fa0a491e05a 00007fa0a4adab3e 00007fa0a4aa8717 00007fa0a4a546ff 00007fa0a4a537be 00007fa0a36ba5c1 00007fa0a36d9d08
1026 02:56:47.187736 (+ 80us) write_op.cc:276] Timestamp: P: 1761447401635168 usec, L: 0
1026 02:56:47.188612 (+ 876us) write_op.cc:270] Start()
1026 02:56:47.188698 (+ 86us) write_op.cc:276] Timestamp: P: 1761447401637161 usec, L: 0
1026 02:56:47.189792 (+ 1094us) write_op.cc:270] Start()
1026 02:56:47.189878 (+ 86us) write_op.cc:276] Timestamp: P: 1761447401638695 usec, L: 0
1026 02:56:47.190485 (+ 607us) write_op.cc:270] S
I20251026 02:56:48.540421 2375 tablet_bootstrap.cc:492] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:48.540558 2375 ts_tablet_manager.cc:1403] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.000s
I20251026 02:56:48.541222 2375 raft_consensus.cc:359] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.541378 2375 raft_consensus.cc:385] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.541420 2375 raft_consensus.cc:740] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.541635 2375 consensus_queue.cc:260] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.542065 2375 ts_tablet_manager.cc:1434] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 02:56:48.542325 2375 tablet_bootstrap.cc:492] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:48.538837 2376 tablet_replica.cc:333] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:48.543385 2376 raft_consensus.cc:2243] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.543557 2376 raft_consensus.cc:2272] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.544042 2375 tablet_bootstrap.cc:654] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.544077 2376 ts_tablet_manager.cc:1916] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.553450 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 966) took 5293 ms. Trace:
I20251026 02:56:48.553586 2372 rpcz_store.cc:276] 1026 02:56:43.260494 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.294887 (+ 34393us) service_pool.cc:225] Handling call
1026 02:56:48.419937 (+5125050us) ts_tablet_manager.cc:1151] Deleting tablet 1fb2ff87cf0248a0a4a77f2733ef867b
1026 02:56:48.419949 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.447417 (+ 27468us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.497542 (+ 50125us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.553419 (+ 55877us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5122937}
I20251026 02:56:48.553730 2381 tablet_bootstrap.cc:492] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:48.553864 2381 ts_tablet_manager.cc:1403] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20251026 02:56:48.553951 2372 tablet_replica.cc:333] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:48.554186 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 1fb2ff87cf0248a0a4a77f2733ef867b (table default.Table_26257eca63ce4baaa75beae2e9db1340 [id=fbc97817a9584474abc0b2ab2052abb2]) successfully deleted
I20251026 02:56:48.554769 2381 raft_consensus.cc:359] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.547745 4040 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: acquiring CA information for follower catalog manager: success
I20251026 02:56:48.557082 2381 raft_consensus.cc:385] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.557240 2381 raft_consensus.cc:740] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.557511 2381 consensus_queue.cc:260] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.558041 2381 ts_tablet_manager.cc:1434] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251026 02:56:48.558399 2381 tablet_bootstrap.cc:492] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:48.560801 4069 raft_consensus.cc:493] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:48.560949 4069 raft_consensus.cc:515] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.561560 4069 leader_election.cc:290] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.562192 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e38f282e8414b7db2a6d5f4a8933e84" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:48.562404 1982 raft_consensus.cc:2468] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:48.562975 2194 leader_election.cc:304] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:48.563256 4069 raft_consensus.cc:2804] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 02:56:48.563326 4069 raft_consensus.cc:493] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:48.563400 4069 raft_consensus.cc:3060] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Advancing to term 2
W20251026 02:56:48.564116 2195 leader_election.cc:343] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 3e38f282e8414b7db2a6d5f4a8933e84
I20251026 02:56:48.564558 2381 tablet_bootstrap.cc:654] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.566037 4003 raft_consensus.cc:493] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.566151 4003 raft_consensus.cc:515] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.566640 4003 leader_election.cc:290] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.567410 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2d6691bb4cf4a8292b0c082e54a91a5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:48.567922 2194 leader_election.cc:343] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.568367 4069 raft_consensus.cc:515] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:48.569092 2195 leader_election.cc:343] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: c2d6691bb4cf4a8292b0c082e54a91a5
I20251026 02:56:48.569192 2195 leader_election.cc:304] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.569446 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e38f282e8414b7db2a6d5f4a8933e84" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:48.569600 4069 leader_election.cc:290] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.569628 1982 raft_consensus.cc:3060] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 2
I20251026 02:56:48.569938 2372 raft_consensus.cc:2243] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.570118 2372 raft_consensus.cc:2272] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.570858 2372 ts_tablet_manager.cc:1916] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.572969 4069 raft_consensus.cc:2749] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.573541 2378 tablet_bootstrap.cc:492] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:48.573676 2378 ts_tablet_manager.cc:1403] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
W20251026 02:56:48.574093 2195 leader_election.cc:343] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 3e38f282e8414b7db2a6d5f4a8933e84
I20251026 02:56:48.574380 2378 raft_consensus.cc:359] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.574544 2378 raft_consensus.cc:385] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.574613 2378 raft_consensus.cc:740] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.574851 1982 raft_consensus.cc:2468] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 2.
I20251026 02:56:48.575347 2194 leader_election.cc:304] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:48.575757 4069 raft_consensus.cc:2804] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 02:56:48.575971 4069 raft_consensus.cc:697] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 2 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:48.576481 4069 consensus_queue.cc:237] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.574819 2378 consensus_queue.cc:260] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.584656 3997 raft_consensus.cc:493] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.584823 3997 raft_consensus.cc:515] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.585392 3997 leader_election.cc:290] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.586038 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66ede86c604e47dbb982e5a3ba18bcb7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:48.586179 2376 ts_tablet_manager.cc:1929] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:48.586310 2376 log.cc:1199] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/ed836bb7519e4b0290e6547fa57128bc
W20251026 02:56:48.586565 2053 leader_election.cc:343] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.586639 2375 tablet_bootstrap.cc:492] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:48.586764 2375 ts_tablet_manager.cc:1403] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.005s
I20251026 02:56:48.587041 2378 ts_tablet_manager.cc:1434] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251026 02:56:48.587229 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66ede86c604e47dbb982e5a3ba18bcb7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:48.587333 2378 tablet_bootstrap.cc:492] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:48.587486 2375 raft_consensus.cc:359] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.587646 2375 raft_consensus.cc:385] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.587692 2375 raft_consensus.cc:740] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
W20251026 02:56:48.587814 2053 leader_election.cc:343] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.587908 2053 leader_election.cc:304] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.587868 2375 consensus_queue.cc:260] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.588445 3997 raft_consensus.cc:2749] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.588879 2378 tablet_bootstrap.cc:654] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.588922 2377 log.cc:1199] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/bfa68f2c942c43e2a02e9d333bdfa2a2
I20251026 02:56:48.589918 2377 ts_tablet_manager.cc:1950] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:48.590485 2376 ts_tablet_manager.cc:1950] T ed836bb7519e4b0290e6547fa57128bc P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:48.591400 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 945) took 3581 ms. Trace:
I20251026 02:56:48.591491 2376 rpcz_store.cc:276] 1026 02:56:45.009979 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.011356 (+ 1377us) service_pool.cc:225] Handling call
1026 02:56:48.538789 (+3527433us) ts_tablet_manager.cc:1151] Deleting tablet ed836bb7519e4b0290e6547fa57128bc
1026 02:56:48.538800 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.569465 (+ 30665us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.586066 (+ 16601us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.591383 (+ 5317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3527178}
I20251026 02:56:48.591940 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet ed836bb7519e4b0290e6547fa57128bc (table default.Table_ec68beb934274468a4f028b3dbaf2cc0 [id=658f4d8e944a4221a722376dc2137896]) successfully deleted
I20251026 02:56:48.592868 2378 tablet_bootstrap.cc:492] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:48.592976 2378 ts_tablet_manager.cc:1403] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20251026 02:56:48.593609 2378 raft_consensus.cc:359] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.593760 2378 raft_consensus.cc:385] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.593801 2378 raft_consensus.cc:740] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.596438 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 969) took 3257 ms. Trace:
I20251026 02:56:48.596552 2376 rpcz_store.cc:276] 1026 02:56:45.338542 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.338617 (+ 75us) service_pool.cc:225] Handling call
1026 02:56:48.596304 (+3257687us) ts_tablet_manager.cc:1151] Deleting tablet 3865399cf6934bc18e16ae2cb4887197
1026 02:56:48.596315 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.596421 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3257434}
W20251026 02:56:48.597090 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 3865399cf6934bc18e16ae2cb4887197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3865399cf6934bc18e16ae2cb4887197 already in progress: creating tablet
I20251026 02:56:48.597537 4040 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 02:56:48.598795 2378 consensus_queue.cc:260] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.599323 2378 ts_tablet_manager.cc:1434] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251026 02:56:48.599579 2378 tablet_bootstrap.cc:492] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:48.599685 4069 raft_consensus.cc:493] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.599886 4069 raft_consensus.cc:515] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.600972 4069 leader_election.cc:290] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.600975 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eabfc95caa84afc9a8956a7ac13fea6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:48.601243 2378 tablet_bootstrap.cc:654] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:48.601608 2194 leader_election.cc:343] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.601611 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eabfc95caa84afc9a8956a7ac13fea6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:48.601805 2116 raft_consensus.cc:2468] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:48.598243 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 970) took 3253 ms. Trace:
I20251026 02:56:48.602217 2195 leader_election.cc:304] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.602288 2376 rpcz_store.cc:276] 1026 02:56:45.345273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.345335 (+ 62us) service_pool.cc:225] Handling call
1026 02:56:48.598131 (+3252796us) ts_tablet_manager.cc:1151] Deleting tablet 6ce4092b7ccc4888b5cbe3b38bee2e36
1026 02:56:48.598140 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.598226 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3252542}
I20251026 02:56:48.602655 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 972) took 3205 ms. Trace:
I20251026 02:56:48.602727 2376 rpcz_store.cc:276] 1026 02:56:45.397579 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.397635 (+ 56us) service_pool.cc:225] Handling call
1026 02:56:48.602544 (+3204909us) ts_tablet_manager.cc:1151] Deleting tablet 94d5faa4584a4e36b38bdd552d641994
1026 02:56:48.602554 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.602641 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3204654}
W20251026 02:56:48.602800 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 already in progress: creating tablet
W20251026 02:56:48.603098 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 94d5faa4584a4e36b38bdd552d641994 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94d5faa4584a4e36b38bdd552d641994 already in progress: creating tablet
I20251026 02:56:48.604862 4069 raft_consensus.cc:2804] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:48.605017 4069 raft_consensus.cc:493] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:48.605144 4069 raft_consensus.cc:3060] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.605206 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 971) took 3212 ms. Trace:
I20251026 02:56:48.605291 2376 rpcz_store.cc:276] 1026 02:56:45.392753 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.437209 (+ 44456us) service_pool.cc:225] Handling call
1026 02:56:48.605077 (+3167868us) ts_tablet_manager.cc:1151] Deleting tablet 4e1291f47f384ff38eadd16040db65b8
1026 02:56:48.605099 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.605189 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3167652}
W20251026 02:56:48.605717 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 4e1291f47f384ff38eadd16040db65b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e1291f47f384ff38eadd16040db65b8 already in progress: creating tablet
I20251026 02:56:48.606452 2376 tablet_replica.cc:333] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:48.606637 2376 raft_consensus.cc:2243] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.606799 2376 raft_consensus.cc:2272] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.607327 2376 ts_tablet_manager.cc:1916] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.609200 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 945) took 5437 ms. Trace:
I20251026 02:56:48.609335 2377 rpcz_store.cc:276] 1026 02:56:43.171396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.171605 (+ 209us) service_pool.cc:225] Handling call
1026 02:56:48.456051 (+5284446us) ts_tablet_manager.cc:1151] Deleting tablet bfa68f2c942c43e2a02e9d333bdfa2a2
1026 02:56:48.456064 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.516834 (+ 60770us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.519972 (+ 3138us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.609181 (+ 89209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5284089}
I20251026 02:56:48.609861 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet bfa68f2c942c43e2a02e9d333bdfa2a2 (table default.Table_f51a5406d89743649dab37ef175a4666 [id=84eea78390b0408dbfeda88cafa9dcd9]) successfully deleted
I20251026 02:56:48.610183 2377 tablet_replica.cc:333] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:48.610397 2377 raft_consensus.cc:2243] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:48.610741 2377 raft_consensus.cc:2272] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.611467 2377 ts_tablet_manager.cc:1916] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.616667 2375 ts_tablet_manager.cc:1434] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.030s user 0.000s sys 0.002s
I20251026 02:56:48.617026 2375 tablet_bootstrap.cc:492] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:48.618790 2375 tablet_bootstrap.cc:654] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.621011 2381 tablet_bootstrap.cc:492] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:48.621289 2381 ts_tablet_manager.cc:1403] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.004s
I20251026 02:56:48.622013 2381 raft_consensus.cc:359] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.622243 2381 raft_consensus.cc:385] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.622351 2381 raft_consensus.cc:740] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.622635 2381 consensus_queue.cc:260] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.623374 2381 ts_tablet_manager.cc:1434] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:48.623713 2381 tablet_bootstrap.cc:492] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:48.625151 2381 tablet_bootstrap.cc:654] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.628103 4069 raft_consensus.cc:515] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.629346 4069 leader_election.cc:290] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.629691 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eabfc95caa84afc9a8956a7ac13fea6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:48.629902 2116 raft_consensus.cc:3060] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.632709 2116 raft_consensus.cc:2468] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:48.633067 2195 leader_election.cc:304] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:48.633147 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eabfc95caa84afc9a8956a7ac13fea6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:48.633812 2194 leader_election.cc:343] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.634169 4003 raft_consensus.cc:2804] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:48.634275 4003 raft_consensus.cc:697] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:48.634524 4003 consensus_queue.cc:237] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.636850 1982 raft_consensus.cc:3060] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.640322 4069 consensus_queue.cc:1048] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:48.648984 4069 raft_consensus.cc:493] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.649245 4069 raft_consensus.cc:515] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.650552 4069 leader_election.cc:290] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.650846 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfd3e23d83a84648bd11cd258377aecb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:48.651057 2116 raft_consensus.cc:2468] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:48.651441 2195 leader_election.cc:304] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:48.651768 4069 raft_consensus.cc:2804] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:48.651877 4069 raft_consensus.cc:493] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:48.651979 4069 raft_consensus.cc:3060] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.652402 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfd3e23d83a84648bd11cd258377aecb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:48.652901 2194 leader_election.cc:343] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.655303 4069 raft_consensus.cc:515] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.655448 2375 tablet_bootstrap.cc:492] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:48.655617 2375 ts_tablet_manager.cc:1403] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.002s
I20251026 02:56:48.656402 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfd3e23d83a84648bd11cd258377aecb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:48.656602 4069 leader_election.cc:290] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.656840 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfd3e23d83a84648bd11cd258377aecb" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
W20251026 02:56:48.656874 2194 leader_election.cc:343] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.656997 2116 raft_consensus.cc:3060] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.657374 2372 ts_tablet_manager.cc:1929] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:48.657490 2372 log.cc:1199] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/4a3f3fc3c7a04d05bae28472251f3acc
I20251026 02:56:48.657775 2372 ts_tablet_manager.cc:1950] T 4a3f3fc3c7a04d05bae28472251f3acc P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:48.658528 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 969) took 5361 ms. Trace:
I20251026 02:56:48.658618 2372 rpcz_store.cc:276] 1026 02:56:43.296898 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.299927 (+ 3029us) service_pool.cc:225] Handling call
1026 02:56:48.553894 (+5253967us) ts_tablet_manager.cc:1151] Deleting tablet 4a3f3fc3c7a04d05bae28472251f3acc
1026 02:56:48.553906 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.620517 (+ 66611us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.657263 (+ 36746us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.658497 (+ 1234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5242157}
I20251026 02:56:48.659092 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 4a3f3fc3c7a04d05bae28472251f3acc (table default.Table_7713a2385fb24bb7bfecfaa78176c128 [id=fae1b7f4879e4d338a4113383721fcc5]) successfully deleted
I20251026 02:56:48.660069 2116 raft_consensus.cc:2468] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:48.660454 2195 leader_election.cc:304] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.656425 2375 raft_consensus.cc:359] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.664258 2375 raft_consensus.cc:385] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.664348 2375 raft_consensus.cc:740] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.665732 2381 tablet_bootstrap.cc:492] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:48.665917 2381 ts_tablet_manager.cc:1403] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20251026 02:56:48.666576 2381 raft_consensus.cc:359] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.666603 2375 consensus_queue.cc:260] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:48.667140 1647 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 51bfa49903d34ff18fb5a0bb9b75295a (table default.Table_6bd4eede457348b0bd4dfe92f0ff146e [id=c13186d1cf86463098c81cfb1dc3b465], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 51bfa49903d34ff18fb5a0bb9b75295a consensus state has no leader
I20251026 02:56:48.668582 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 971) took 5364 ms. Trace:
I20251026 02:56:48.668670 2372 rpcz_store.cc:276] 1026 02:56:43.304106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.396750 (+ 92644us) service_pool.cc:225] Handling call
1026 02:56:48.658910 (+5262160us) ts_tablet_manager.cc:1151] Deleting tablet 035724949fe142ea93cb998f8fdc4344
1026 02:56:48.658919 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.668561 (+ 9642us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5261922}
W20251026 02:56:48.669411 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 035724949fe142ea93cb998f8fdc4344 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 035724949fe142ea93cb998f8fdc4344 already in progress: creating tablet
I20251026 02:56:48.668871 2375 ts_tablet_manager.cc:1434] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251026 02:56:48.670284 2375 tablet_bootstrap.cc:492] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:48.671654 2376 ts_tablet_manager.cc:1929] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.671972 2375 tablet_bootstrap.cc:654] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.671981 2381 raft_consensus.cc:385] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.672084 2381 raft_consensus.cc:740] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.672369 2381 consensus_queue.cc:260] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.672704 2376 log.cc:1199] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/bfa68f2c942c43e2a02e9d333bdfa2a2
I20251026 02:56:48.673511 2381 ts_tablet_manager.cc:1434] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251026 02:56:48.674213 2376 ts_tablet_manager.cc:1950] T bfa68f2c942c43e2a02e9d333bdfa2a2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:48.674552 2381 tablet_bootstrap.cc:492] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:48.675534 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 974) took 3157 ms. Trace:
I20251026 02:56:48.676817 4098 raft_consensus.cc:493] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.676976 4098 raft_consensus.cc:515] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.677196 2381 tablet_bootstrap.cc:654] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.668975 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 963) took 5426 ms. Trace:
I20251026 02:56:48.677592 2372 rpcz_store.cc:276] 1026 02:56:43.242178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.400383 (+158205us) service_pool.cc:225] Handling call
1026 02:56:48.668899 (+5268516us) ts_tablet_manager.cc:1151] Deleting tablet 0245af5a2dcb4f569b62307965895c1f
1026 02:56:48.668907 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.668965 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5268275}
I20251026 02:56:48.677673 4098 leader_election.cc:290] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.676757 2378 tablet_bootstrap.cc:492] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:48.677917 2378 ts_tablet_manager.cc:1403] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.078s user 0.001s sys 0.005s
I20251026 02:56:48.677954 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 975) took 4854 ms. Trace:
I20251026 02:56:48.678009 2372 rpcz_store.cc:276] 1026 02:56:43.823404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.823461 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:48.677839 (+4854378us) ts_tablet_manager.cc:1151] Deleting tablet bcd5621aa3754733b22508be89f080db
1026 02:56:48.677848 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.677939 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4854185}
I20251026 02:56:48.678336 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 977) took 4852 ms. Trace:
I20251026 02:56:48.678395 2372 rpcz_store.cc:276] 1026 02:56:43.826062 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.826572 (+ 510us) service_pool.cc:225] Handling call
1026 02:56:48.678248 (+4851676us) ts_tablet_manager.cc:1151] Deleting tablet 8fdadd053cc34aeca0d491eb3200207f
1026 02:56:48.678256 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.678324 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4851511}
I20251026 02:56:48.678665 2372 tablet_replica.cc:333] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:48.678637 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0245af5a2dcb4f569b62307965895c1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0245af5a2dcb4f569b62307965895c1f already in progress: creating tablet
I20251026 02:56:48.678831 2372 raft_consensus.cc:2243] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.678812 2378 raft_consensus.cc:359] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.678956 2378 raft_consensus.cc:385] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.678953 2372 raft_consensus.cc:2272] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.679001 2378 raft_consensus.cc:740] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
W20251026 02:56:48.678993 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet bcd5621aa3754733b22508be89f080db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcd5621aa3754733b22508be89f080db already in progress: creating tablet
I20251026 02:56:48.679193 2378 consensus_queue.cc:260] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.679435 2372 ts_tablet_manager.cc:1916] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.680151 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ac38e6257e74367b06b15fc49d2af81" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:48.680429 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 8fdadd053cc34aeca0d491eb3200207f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fdadd053cc34aeca0d491eb3200207f already in progress: creating tablet
I20251026 02:56:48.679731 2378 ts_tablet_manager.cc:1434] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 02:56:48.680918 2378 tablet_bootstrap.cc:492] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:48.680930 2053 leader_election.cc:343] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.681423 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ac38e6257e74367b06b15fc49d2af81" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:48.682052 2053 leader_election.cc:343] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.682121 2377 ts_tablet_manager.cc:1929] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.682214 2377 log.cc:1199] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/21b936fa4f6a4ff6af321971a30cccbe
I20251026 02:56:48.682184 2053 leader_election.cc:304] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.682451 2378 tablet_bootstrap.cc:654] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.682543 2377 ts_tablet_manager.cc:1950] T 21b936fa4f6a4ff6af321971a30cccbe P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:48.682757 3997 raft_consensus.cc:2749] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.683228 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 948) took 5511 ms. Trace:
I20251026 02:56:48.683332 2376 rpcz_store.cc:276] 1026 02:56:45.518285 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.526558 (+ 8273us) service_pool.cc:225] Handling call
1026 02:56:48.606407 (+3079849us) ts_tablet_manager.cc:1151] Deleting tablet bfa68f2c942c43e2a02e9d333bdfa2a2
1026 02:56:48.606417 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.626772 (+ 20355us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.671515 (+ 44743us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.675510 (+ 3995us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3071180}
I20251026 02:56:48.683712 2376 tablet_replica.cc:333] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:48.683926 2376 raft_consensus.cc:2243] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:48.684175 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet bfa68f2c942c43e2a02e9d333bdfa2a2 (table default.Table_f51a5406d89743649dab37ef175a4666 [id=84eea78390b0408dbfeda88cafa9dcd9]) successfully deleted
I20251026 02:56:48.684335 2376 raft_consensus.cc:2272] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.687254 2376 ts_tablet_manager.cc:1916] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.697286 2377 rpcz_store.cc:276] 1026 02:56:43.171645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.172905 (+ 1260us) service_pool.cc:225] Handling call
1026 02:56:48.610139 (+5437234us) ts_tablet_manager.cc:1151] Deleting tablet 21b936fa4f6a4ff6af321971a30cccbe
1026 02:56:48.610148 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.662152 (+ 52004us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.682018 (+ 19866us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.683210 (+ 1192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5437106}
I20251026 02:56:48.697952 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 21b936fa4f6a4ff6af321971a30cccbe (table default.Table_4e2e1ce85b8246388602db5ddc857f79 [id=5fb61e88f7074976bac29b40f4466ecb]) successfully deleted
I20251026 02:56:48.698282 2378 tablet_bootstrap.cc:492] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:48.698422 2378 ts_tablet_manager.cc:1403] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20251026 02:56:48.669049 4069 raft_consensus.cc:2804] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:48.690299 4067 raft_consensus.cc:493] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.699132 2378 raft_consensus.cc:359] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.699306 2378 raft_consensus.cc:385] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.699357 2378 raft_consensus.cc:740] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.698668 4067 raft_consensus.cc:515] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.699568 2378 consensus_queue.cc:260] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.700054 4067 leader_election.cc:290] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.700147 2378 ts_tablet_manager.cc:1434] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:48.700436 2378 tablet_bootstrap.cc:492] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:48.698777 4069 raft_consensus.cc:697] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:48.701078 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a1442fffe54add93666e3c78a0cc58" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:48.701316 2116 raft_consensus.cc:2468] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:48.701725 2195 leader_election.cc:304] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:48.702019 4067 raft_consensus.cc:2804] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:48.702098 4067 raft_consensus.cc:493] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:48.702170 2378 tablet_bootstrap.cc:654] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.702165 4067 raft_consensus.cc:3060] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.702108 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a1442fffe54add93666e3c78a0cc58" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:48.704669 2194 leader_election.cc:343] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.705535 4067 raft_consensus.cc:515] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.707044 1982 raft_consensus.cc:3060] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.707648 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a1442fffe54add93666e3c78a0cc58" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:48.707844 4067 leader_election.cc:290] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.707888 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a1442fffe54add93666e3c78a0cc58" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
W20251026 02:56:48.708154 2194 leader_election.cc:343] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.708339 2116 raft_consensus.cc:3060] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.701462 4069 consensus_queue.cc:237] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.708460 2381 tablet_bootstrap.cc:492] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:48.708612 2381 ts_tablet_manager.cc:1403] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.001s
I20251026 02:56:48.698464 2377 tablet_replica.cc:333] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:48.711275 2381 raft_consensus.cc:359] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.711537 2381 raft_consensus.cc:385] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.711519 2116 raft_consensus.cc:2468] T 41a1442fffe54add93666e3c78a0cc58 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:48.711647 2381 raft_consensus.cc:740] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.711861 2195 leader_election.cc:304] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.711915 2381 consensus_queue.cc:260] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.712291 4003 raft_consensus.cc:2804] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:48.712442 2381 ts_tablet_manager.cc:1434] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251026 02:56:48.712576 4003 raft_consensus.cc:697] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:48.712719 2381 tablet_bootstrap.cc:492] T e203fe0403594d30b44d531153d42a10 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:48.712821 4003 consensus_queue.cc:237] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.714624 2381 tablet_bootstrap.cc:654] T e203fe0403594d30b44d531153d42a10 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.716107 2376 ts_tablet_manager.cc:1929] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.715169 4067 consensus_queue.cc:1048] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:48.717785 3997 raft_consensus.cc:493] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.717916 3997 raft_consensus.cc:515] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.718039 2377 raft_consensus.cc:2243] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 2 LEADER]: Raft consensus shutting down.
I20251026 02:56:48.718263 2377 pending_rounds.cc:70] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:48.718300 2377 pending_rounds.cc:77] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7214888585531830272 op_type: NO_OP noop_request { }
I20251026 02:56:48.718365 2377 raft_consensus.cc:2889] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:48.718425 2377 raft_consensus.cc:2272] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.718506 3997 leader_election.cc:290] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.719233 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31b6e08597bd450eb1875201cf594a82" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:48.719806 2053 leader_election.cc:343] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.720173 2377 ts_tablet_manager.cc:1916] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.719321 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31b6e08597bd450eb1875201cf594a82" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:48.720305 2378 tablet_bootstrap.cc:492] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:48.720633 2378 ts_tablet_manager.cc:1403] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
W20251026 02:56:48.720738 2053 leader_election.cc:343] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.720836 2053 leader_election.cc:304] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.721273 3997 raft_consensus.cc:2749] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.721395 2378 raft_consensus.cc:359] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.721566 2378 raft_consensus.cc:385] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.721637 2378 raft_consensus.cc:740] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.721824 2376 log.cc:1199] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/d212170f056a44dfb15cfcb7f9fbae2f
I20251026 02:56:48.722193 2376 ts_tablet_manager.cc:1950] T d212170f056a44dfb15cfcb7f9fbae2f P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:48.722906 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 973) took 3205 ms. Trace:
I20251026 02:56:48.723024 2376 rpcz_store.cc:276] 1026 02:56:45.517907 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.526809 (+ 8902us) service_pool.cc:225] Handling call
1026 02:56:48.683654 (+3156845us) ts_tablet_manager.cc:1151] Deleting tablet d212170f056a44dfb15cfcb7f9fbae2f
1026 02:56:48.683667 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.690279 (+ 6612us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.715959 (+ 25680us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.722872 (+ 6913us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3148351}
I20251026 02:56:48.723474 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet d212170f056a44dfb15cfcb7f9fbae2f (table default.Table_7667f4f99bf744579f1dc2b2d0f469eb [id=822eea61b89846f4828511c58a9da4c8]) successfully deleted
I20251026 02:56:48.725706 4167 raft_consensus.cc:493] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.725840 4167 raft_consensus.cc:515] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.726353 3997 raft_consensus.cc:493] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.726442 4167 leader_election.cc:290] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.726470 3997 raft_consensus.cc:515] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.726845 2378 consensus_queue.cc:260] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.726984 3997 leader_election.cc:290] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.727011 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cf76ef8f6af445eb0ce15c8d79136c5" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:48.727070 1916 leader_election.cc:343] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 0cf76ef8f6af445eb0ce15c8d79136c5
I20251026 02:56:48.727172 2259 raft_consensus.cc:2468] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:48.727561 1915 leader_election.cc:304] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:48.727717 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109073eb664d4090b7a2b786146e13fa" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:48.727977 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109073eb664d4090b7a2b786146e13fa" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:48.728260 2053 leader_election.cc:343] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.728305 4167 raft_consensus.cc:2804] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:48.728377 4167 raft_consensus.cc:493] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:48.728454 4167 raft_consensus.cc:3060] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:48.728473 2053 leader_election.cc:343] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.728557 2053 leader_election.cc:304] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.728794 3997 raft_consensus.cc:2749] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.731519 4167 raft_consensus.cc:515] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.732312 4167 leader_election.cc:290] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.732630 2376 tablet_replica.cc:333] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:48.732597 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cf76ef8f6af445eb0ce15c8d79136c5" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:48.732753 2259 raft_consensus.cc:3060] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.733040 2376 raft_consensus.cc:2243] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.733191 2376 raft_consensus.cc:2272] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.732813 2378 ts_tablet_manager.cc:1434] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251026 02:56:48.734004 2378 tablet_bootstrap.cc:492] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:48.735373 2259 raft_consensus.cc:2468] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:48.735565 2378 tablet_bootstrap.cc:654] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.736395 2376 ts_tablet_manager.cc:1916] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.737251 1915 leader_election.cc:304] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:48.737618 4167 raft_consensus.cc:2804] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 02:56:48.737733 1916 leader_election.cc:343] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 0cf76ef8f6af445eb0ce15c8d79136c5
I20251026 02:56:48.737948 4167 raft_consensus.cc:697] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
W20251026 02:56:48.738225 2195 consensus_peers.cc:597] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a3f3fc3c7a04d05bae28472251f3acc. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:48.739089 4167 consensus_queue.cc:237] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.742116 2372 ts_tablet_manager.cc:1929] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:48.742235 2372 log.cc:1199] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/d43b6f22dd4f4f9c9657f21514550312
I20251026 02:56:48.742558 2372 ts_tablet_manager.cc:1950] T d43b6f22dd4f4f9c9657f21514550312 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:48.743306 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 980) took 4830 ms. Trace:
I20251026 02:56:48.743395 2372 rpcz_store.cc:276] 1026 02:56:43.912397 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.912466 (+ 69us) service_pool.cc:225] Handling call
1026 02:56:48.678626 (+4766160us) ts_tablet_manager.cc:1151] Deleting tablet d43b6f22dd4f4f9c9657f21514550312
1026 02:56:48.678634 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.710960 (+ 32326us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.742007 (+ 31047us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.743288 (+ 1281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4762517}
I20251026 02:56:48.743788 2372 tablet_replica.cc:333] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:48.744016 2372 raft_consensus.cc:2243] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.744038 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet d43b6f22dd4f4f9c9657f21514550312 (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
I20251026 02:56:48.744168 2372 raft_consensus.cc:2272] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.747959 2377 ts_tablet_manager.cc:1929] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251026 02:56:48.749991 2378 tablet_bootstrap.cc:492] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:48.750149 2378 ts_tablet_manager.cc:1403] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20251026 02:56:48.750872 2378 raft_consensus.cc:359] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.751044 2378 raft_consensus.cc:385] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.751137 2378 raft_consensus.cc:740] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.751401 2378 consensus_queue.cc:260] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.754415 2378 ts_tablet_manager.cc:1434] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 02:56:48.754732 2378 tablet_bootstrap.cc:492] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:48.756376 2378 tablet_bootstrap.cc:654] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.757573 2372 ts_tablet_manager.cc:1916] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.748988 2381 tablet_bootstrap.cc:492] T e203fe0403594d30b44d531153d42a10 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:48.759819 2377 log.cc:1199] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3e38f282e8414b7db2a6d5f4a8933e84
I20251026 02:56:48.759877 2381 ts_tablet_manager.cc:1403] T e203fe0403594d30b44d531153d42a10 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20251026 02:56:48.760435 2377 ts_tablet_manager.cc:1950] T 3e38f282e8414b7db2a6d5f4a8933e84 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:48.757647 2375 tablet_bootstrap.cc:492] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:48.760697 2375 ts_tablet_manager.cc:1403] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.090s user 0.011s sys 0.031s
I20251026 02:56:48.760818 2381 raft_consensus.cc:359] T e203fe0403594d30b44d531153d42a10 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.761059 2381 raft_consensus.cc:385] T e203fe0403594d30b44d531153d42a10 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.761178 2381 raft_consensus.cc:740] T e203fe0403594d30b44d531153d42a10 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.761482 2375 raft_consensus.cc:359] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.761667 2375 raft_consensus.cc:385] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.761463 2381 consensus_queue.cc:260] T e203fe0403594d30b44d531153d42a10 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.761770 2375 raft_consensus.cc:740] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.761963 2375 consensus_queue.cc:260] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.762427 2381 ts_tablet_manager.cc:1434] T e203fe0403594d30b44d531153d42a10 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:48.762445 2375 ts_tablet_manager.cc:1434] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:48.762728 2375 tablet_bootstrap.cc:492] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:48.762750 2381 tablet_bootstrap.cc:492] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:48.764359 2381 tablet_bootstrap.cc:654] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.764575 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 949) took 5592 ms. Trace:
I20251026 02:56:48.764747 2375 tablet_bootstrap.cc:654] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.764838 2377 rpcz_store.cc:276] 1026 02:56:43.171726 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.173266 (+ 1540us) service_pool.cc:225] Handling call
1026 02:56:48.698405 (+5525139us) ts_tablet_manager.cc:1151] Deleting tablet 3e38f282e8414b7db2a6d5f4a8933e84
1026 02:56:48.698418 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.744961 (+ 46543us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.747874 (+ 2913us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.764553 (+ 16679us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2826,"tablet-delete.queue_time_us":5524949}
I20251026 02:56:48.766342 2376 ts_tablet_manager.cc:1929] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.766368 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3e38f282e8414b7db2a6d5f4a8933e84 (table default.Table_3a75d3c1b8304d1bbcc19c12111f24dd [id=e2f6dfa3686b41ddbc9f3b0205c7f05f]) successfully deleted
I20251026 02:56:48.766492 2376 log.cc:1199] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/21b936fa4f6a4ff6af321971a30cccbe
I20251026 02:56:48.766870 2377 tablet_replica.cc:333] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:48.766929 2376 ts_tablet_manager.cc:1950] T 21b936fa4f6a4ff6af321971a30cccbe P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:48.767508 2377 raft_consensus.cc:2243] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:48.767733 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 976) took 3221 ms. Trace:
I20251026 02:56:48.767835 2376 rpcz_store.cc:276] 1026 02:56:45.546734 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.568384 (+ 21650us) service_pool.cc:225] Handling call
1026 02:56:48.732566 (+3164182us) ts_tablet_manager.cc:1151] Deleting tablet 21b936fa4f6a4ff6af321971a30cccbe
1026 02:56:48.732576 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.745919 (+ 13343us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.766184 (+ 20265us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.767688 (+ 1504us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3145781}
I20251026 02:56:48.768437 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 21b936fa4f6a4ff6af321971a30cccbe (table default.Table_4e2e1ce85b8246388602db5ddc857f79 [id=5fb61e88f7074976bac29b40f4466ecb]) successfully deleted
I20251026 02:56:48.768200 2376 tablet_replica.cc:333] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:48.773103 2376 raft_consensus.cc:2243] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.773167 2377 raft_consensus.cc:2272] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.779824 2375 tablet_bootstrap.cc:492] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:48.779981 2375 ts_tablet_manager.cc:1403] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.006s
I20251026 02:56:48.780778 2375 raft_consensus.cc:359] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.780941 2375 raft_consensus.cc:385] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.780995 2375 raft_consensus.cc:740] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.781196 2375 consensus_queue.cc:260] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.781687 2375 ts_tablet_manager.cc:1434] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:48.781953 2375 tablet_bootstrap.cc:492] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:48.783594 2375 tablet_bootstrap.cc:654] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.789618 2376 raft_consensus.cc:2272] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.790269 2376 ts_tablet_manager.cc:1916] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.793696 2375 tablet_bootstrap.cc:492] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:48.793875 2375 ts_tablet_manager.cc:1403] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20251026 02:56:48.794693 2375 raft_consensus.cc:359] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.794854 2375 raft_consensus.cc:385] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.794901 2375 raft_consensus.cc:740] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.795102 2375 consensus_queue.cc:260] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.795717 2375 ts_tablet_manager.cc:1434] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:48.795995 2375 tablet_bootstrap.cc:492] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:48.796011 2378 tablet_bootstrap.cc:492] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:48.796154 2378 ts_tablet_manager.cc:1403] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.004s
I20251026 02:56:48.797673 2375 tablet_bootstrap.cc:654] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.801605 2375 tablet_bootstrap.cc:492] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:48.801753 2375 ts_tablet_manager.cc:1403] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
W20251026 02:56:48.801940 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91f81d156ff44decb63a4c0f5eacdac9 consensus state has no leader
I20251026 02:56:48.802325 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_342101e3599e42358d22806e5e119290"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:48.802503 2375 raft_consensus.cc:359] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.802646 2375 raft_consensus.cc:385] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.802690 2375 raft_consensus.cc:740] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
W20251026 02:56:48.802968 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_342101e3599e42358d22806e5e119290 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.802886 2375 consensus_queue.cc:260] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.803570 2375 ts_tablet_manager.cc:1434] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:48.803860 2375 tablet_bootstrap.cc:492] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:48.803889 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet b2eab97dc2b9409697589a34e9cc7d6c from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:48.804337 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 102944a3ae83481089df88fc6b8388af consensus state has no leader
W20251026 02:56:48.804597 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e0b929f603b4c25b5b971ee9db683e8 consensus state has no leader
I20251026 02:56:48.804668 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet 19016e8e11cc42c69ce44a819db9e298 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:48.797576 2378 raft_consensus.cc:359] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.804899 2378 raft_consensus.cc:385] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.804924 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b17021b7e5d845748cfd8d145b9721ce" } modify_external_catalogs: true
I20251026 02:56:48.804986 2378 raft_consensus.cc:740] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.805146 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b17021b7e5d845748cfd8d145b9721ce" } modify_external_catalogs: true
I20251026 02:56:48.805296 2378 consensus_queue.cc:260] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.806138 2375 tablet_bootstrap.cc:654] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.807224 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 7834a85168364390a841fc883681c845 (table default.Table_42b54262967149fda8be1e83c4579b5d [id=acf34beac75d4537a2af3b818886880c]). Expected version 1 got 0
I20251026 02:56:48.807426 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 386e1bd4635b4a7e8e7f249b577fd978 (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]). Expected version 2 got 0
I20251026 02:56:48.809309 4098 raft_consensus.cc:493] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.809437 4098 raft_consensus.cc:515] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.810007 4098 leader_election.cc:290] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:48.811153 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18244445760a434ba7ab2f3a974da7a0 consensus state has no leader
I20251026 02:56:48.811535 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a263fecc382f46039c6cdad42a49517d" } modify_external_catalogs: true
I20251026 02:56:48.811695 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a263fecc382f46039c6cdad42a49517d" } modify_external_catalogs: true
I20251026 02:56:48.812657 2378 ts_tablet_manager.cc:1434] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20251026 02:56:48.813129 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 4c80e9771e734651a3f393829df62de7 (DEFAULT_TABLE table=default.Table_6b4fcf2a2bad4ca7ae3d9bce7d17c222 [id=2fd6bcec67c14b99a748a85ff4d5d85b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.813565 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c80e9771e734651a3f393829df62de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.817592 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "398c1f420974453b9b6b0fe17795378b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:48.818305 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "398c1f420974453b9b6b0fe17795378b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:48.818439 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 4c80e9771e734651a3f393829df62de7 (DEFAULT_TABLE table=default.Table_6b4fcf2a2bad4ca7ae3d9bce7d17c222 [id=2fd6bcec67c14b99a748a85ff4d5d85b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.818727 2378 tablet_bootstrap.cc:492] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:48.818742 2375 tablet_bootstrap.cc:492] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
W20251026 02:56:48.818818 2053 leader_election.cc:343] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.818868 2375 ts_tablet_manager.cc:1403] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
W20251026 02:56:48.819026 2053 leader_election.cc:343] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.819114 2053 leader_election.cc:304] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.819746 4098 raft_consensus.cc:2749] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.820040 2375 raft_consensus.cc:359] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.820202 2375 raft_consensus.cc:385] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:48.820391 2375 raft_consensus.cc:740] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:48.820488 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c80e9771e734651a3f393829df62de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.820626 2375 consensus_queue.cc:260] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.821210 2375 ts_tablet_manager.cc:1434] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 02:56:48.821470 2375 tablet_bootstrap.cc:492] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:48.823658 1656 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 109073eb664d4090b7a2b786146e13fa (table default.Table_9c224d9efebb4e9ea1f3b2b2bb63335e [id=da72d631aee04d10a6832b04f208587f]). Expected version 4 got 0
I20251026 02:56:48.823985 2375 tablet_bootstrap.cc:654] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:48.827208 1654 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 2fd8f02126b24285bbfa575540ba0f09 (table default.Table_7539d36a088f4eafaa21d151ed9ea1cb [id=1950d7cb826b4a99b1ba8866e1d845f3]). Expected version 1 got 0
I20251026 02:56:48.828923 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 4c80e9771e734651a3f393829df62de7 (DEFAULT_TABLE table=default.Table_6b4fcf2a2bad4ca7ae3d9bce7d17c222 [id=2fd6bcec67c14b99a748a85ff4d5d85b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.829358 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c80e9771e734651a3f393829df62de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.829541 2232 tablet_service.cc:1505] Processing CreateTablet for tablet ad1499eaa7a14a53afcc670bd00e8032 (DEFAULT_TABLE table=default.Table_9b4d5b6ca1484e499853db57b4a4c416 [id=76c4695dc0344f1e8dbaee8fbdb42067]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.829891 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad1499eaa7a14a53afcc670bd00e8032. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.830052 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 842740d1b30a49769a2fa77a2813c1ae (DEFAULT_TABLE table=default.Table_3cf2d84ee15542b888e863fa40b143fb [id=7a4947999d9449409f6db5150227691f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.830560 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 842740d1b30a49769a2fa77a2813c1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.831480 2376 ts_tablet_manager.cc:1929] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:48.831591 2376 log.cc:1199] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/f437d8b5f5784e8fbec10d0700e90348
I20251026 02:56:48.831986 2376 ts_tablet_manager.cc:1950] T f437d8b5f5784e8fbec10d0700e90348 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:48.832809 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 975) took 3287 ms. Trace:
I20251026 02:56:48.832918 2376 rpcz_store.cc:276] 1026 02:56:45.545557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.568900 (+ 23343us) service_pool.cc:225] Handling call
1026 02:56:48.768120 (+3199220us) ts_tablet_manager.cc:1151] Deleting tablet f437d8b5f5784e8fbec10d0700e90348
1026 02:56:48.768131 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.819748 (+ 51617us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.831354 (+ 11606us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.832766 (+ 1412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3176200}
I20251026 02:56:48.833034 2100 tablet_service.cc:1505] Processing CreateTablet for tablet ad1499eaa7a14a53afcc670bd00e8032 (DEFAULT_TABLE table=default.Table_9b4d5b6ca1484e499853db57b4a4c416 [id=76c4695dc0344f1e8dbaee8fbdb42067]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.833372 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad1499eaa7a14a53afcc670bd00e8032. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.833382 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 842740d1b30a49769a2fa77a2813c1ae (DEFAULT_TABLE table=default.Table_3cf2d84ee15542b888e863fa40b143fb [id=7a4947999d9449409f6db5150227691f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.833710 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 842740d1b30a49769a2fa77a2813c1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.836030 1948 tablet_service.cc:1505] Processing CreateTablet for tablet ad1499eaa7a14a53afcc670bd00e8032 (DEFAULT_TABLE table=default.Table_9b4d5b6ca1484e499853db57b4a4c416 [id=76c4695dc0344f1e8dbaee8fbdb42067]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.836448 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad1499eaa7a14a53afcc670bd00e8032. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.836870 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 842740d1b30a49769a2fa77a2813c1ae (DEFAULT_TABLE table=default.Table_3cf2d84ee15542b888e863fa40b143fb [id=7a4947999d9449409f6db5150227691f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.836876 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 8e7965fd3a164abb8fb4642e731190f7 (DEFAULT_TABLE table=default.Table_b7162d7bfb2b4b16b958aa22d192d63c [id=00c0f862c8a442f58fe87a08c26683ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.837175 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 842740d1b30a49769a2fa77a2813c1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.837206 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e7965fd3a164abb8fb4642e731190f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.840904 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet f437d8b5f5784e8fbec10d0700e90348 (table default.Table_b555f404492a47808d8a9a168a45ef2f [id=6185bafc43f340dfa6e187a6e5468874]) successfully deleted
I20251026 02:56:48.841815 1663 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 386e1bd4635b4a7e8e7f249b577fd978 (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]). Expected version 2 got 0
I20251026 02:56:48.842060 1663 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 2fd8f02126b24285bbfa575540ba0f09 (table default.Table_7539d36a088f4eafaa21d151ed9ea1cb [id=1950d7cb826b4a99b1ba8866e1d845f3]). Expected version 1 got 0
I20251026 02:56:48.847419 2376 tablet_replica.cc:333] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:48.847651 2376 raft_consensus.cc:2243] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.848771 2376 raft_consensus.cc:2272] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.861791 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 8e7965fd3a164abb8fb4642e731190f7 (DEFAULT_TABLE table=default.Table_b7162d7bfb2b4b16b958aa22d192d63c [id=00c0f862c8a442f58fe87a08c26683ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.862224 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e7965fd3a164abb8fb4642e731190f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.862697 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet abf58185766448d1a7c4f3d304b0c5b1
I20251026 02:56:48.863394 1660 catalog_manager.cc:6719] ReplaceTablet: tablet abf58185766448d1a7c4f3d304b0c5b1 deleted and replaced by tablet 41cd60fe93d44cc4b07789a6288d86d9
I20251026 02:56:48.863953 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_843ed42261d54cfb917339822d4b30dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:48.864398 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet b2eab97dc2b9409697589a34e9cc7d6c
I20251026 02:56:48.864782 1661 catalog_manager.cc:6719] ReplaceTablet: tablet b2eab97dc2b9409697589a34e9cc7d6c deleted and replaced by tablet 1e9482cdd69647dab6bcc2e1c75e4650
I20251026 02:56:48.865051 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 163b8f785675474e90ff90a1fe6fc056
I20251026 02:56:48.865126 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5710a38a4bd74f5a8a6818bbd7bcae5f" } modify_external_catalogs: true
I20251026 02:56:48.865298 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5710a38a4bd74f5a8a6818bbd7bcae5f" } modify_external_catalogs: true
I20251026 02:56:48.865948 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5e627b2d763041a788ce88beb39f25d6" } modify_external_catalogs: true
I20251026 02:56:48.866178 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5e627b2d763041a788ce88beb39f25d6" } modify_external_catalogs: true
I20251026 02:56:48.866746 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 19016e8e11cc42c69ce44a819db9e298
I20251026 02:56:48.867156 1659 catalog_manager.cc:6719] ReplaceTablet: tablet 19016e8e11cc42c69ce44a819db9e298 deleted and replaced by tablet 46f28f2f55ac41249599e4b9a4a53890
I20251026 02:56:48.867344 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet 12e10fd463cd43c7abb22ef428c0aa37 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:48.868577 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 551cc0916a21463d9190d14a31e79988
I20251026 02:56:48.868803 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 19016e8e11cc42c69ce44a819db9e298 with delete_type TABLET_DATA_DELETED (replaced by tablet 46f28f2f55ac41249599e4b9a4a53890) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.868849 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 163b8f785675474e90ff90a1fe6fc056 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.868803 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet abf58185766448d1a7c4f3d304b0c5b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41cd60fe93d44cc4b07789a6288d86d9) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.869328 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a1d0a91306ea4540bbceec66fe7abb8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:48.869546 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3597ea2076244c4ba6e67599f245a639"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:48.870018 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1d0a91306ea4540bbceec66fe7abb8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:48.870226 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3597ea2076244c4ba6e67599f245a639 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.871243 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet b2eab97dc2b9409697589a34e9cc7d6c with delete_type TABLET_DATA_DELETED (replaced by tablet 1e9482cdd69647dab6bcc2e1c75e4650) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.872005 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 551cc0916a21463d9190d14a31e79988 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:48.873169 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_843ed42261d54cfb917339822d4b30dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.875950 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet abf58185766448d1a7c4f3d304b0c5b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41cd60fe93d44cc4b07789a6288d86d9) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.876111 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 163b8f785675474e90ff90a1fe6fc056 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.876138 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet b2eab97dc2b9409697589a34e9cc7d6c with delete_type TABLET_DATA_DELETED (replaced by tablet 1e9482cdd69647dab6bcc2e1c75e4650) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.876277 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 19016e8e11cc42c69ce44a819db9e298 with delete_type TABLET_DATA_DELETED (replaced by tablet 46f28f2f55ac41249599e4b9a4a53890) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.877144 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 551cc0916a21463d9190d14a31e79988 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.877399 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet abf58185766448d1a7c4f3d304b0c5b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41cd60fe93d44cc4b07789a6288d86d9) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.877498 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet b2eab97dc2b9409697589a34e9cc7d6c with delete_type TABLET_DATA_DELETED (replaced by tablet 1e9482cdd69647dab6bcc2e1c75e4650) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.877677 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 19016e8e11cc42c69ce44a819db9e298 with delete_type TABLET_DATA_DELETED (replaced by tablet 46f28f2f55ac41249599e4b9a4a53890) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.877717 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 163b8f785675474e90ff90a1fe6fc056 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.877889 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 551cc0916a21463d9190d14a31e79988 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.883291 4167 raft_consensus.cc:493] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.883442 4167 raft_consensus.cc:515] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.884033 4167 leader_election.cc:290] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:48.884776 1916 leader_election.cc:343] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9a345fa5edbd48d2891b8b40b09cb3f8
I20251026 02:56:48.885347 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a345fa5edbd48d2891b8b40b09cb3f8" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:48.885567 2259 raft_consensus.cc:2468] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:48.885931 1915 leader_election.cc:304] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:48.886071 4098 raft_consensus.cc:493] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.886184 4098 raft_consensus.cc:515] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.886320 4167 raft_consensus.cc:2804] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:48.886381 4167 raft_consensus.cc:493] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:48.886435 4167 raft_consensus.cc:3060] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.886762 4098 leader_election.cc:290] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.887560 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3af2695c41a4439bc4092b7cb1b2edf" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:48.888006 2053 leader_election.cc:343] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.888417 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3af2695c41a4439bc4092b7cb1b2edf" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:48.888912 2053 leader_election.cc:343] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.889004 2053 leader_election.cc:304] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.888975 4167 raft_consensus.cc:515] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.889358 4098 raft_consensus.cc:2749] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.889583 4167 leader_election.cc:290] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:48.890326 1916 leader_election.cc:343] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9a345fa5edbd48d2891b8b40b09cb3f8
I20251026 02:56:48.890374 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a345fa5edbd48d2891b8b40b09cb3f8" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:48.890553 2259 raft_consensus.cc:3060] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.893427 2259 raft_consensus.cc:2468] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
W20251026 02:56:48.893442 1916 consensus_peers.cc:597] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62d30d79a6e4432a8dbe40a104e87fa0. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:48.893987 1915 leader_election.cc:304] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:48.894395 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 662a86b2a3af4ad4b5edee9e5d6ff90e (DEFAULT_TABLE table=default.Table_a08b6588e1414d00874ff1ec6c01c00b [id=8d288e42a17e4bf981c8b360a5d55c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.894465 4167 raft_consensus.cc:2804] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:48.894822 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 662a86b2a3af4ad4b5edee9e5d6ff90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.895133 4167 raft_consensus.cc:697] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:48.895398 4167 consensus_queue.cc:237] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.901562 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 8e7965fd3a164abb8fb4642e731190f7 (DEFAULT_TABLE table=default.Table_b7162d7bfb2b4b16b958aa22d192d63c [id=00c0f862c8a442f58fe87a08c26683ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.902143 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e7965fd3a164abb8fb4642e731190f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.906805 2084 tablet_service.cc:1505] Processing CreateTablet for tablet fecdad6d120541bd8d3990cbd9b0a982 (DEFAULT_TABLE table=default.Table_7549c6a144774b478402b14065800d6b [id=6255d790c1f7481c979bdb4c00f08a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.907238 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fecdad6d120541bd8d3990cbd9b0a982. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.907593 2232 tablet_service.cc:1505] Processing CreateTablet for tablet fecdad6d120541bd8d3990cbd9b0a982 (DEFAULT_TABLE table=default.Table_7549c6a144774b478402b14065800d6b [id=6255d790c1f7481c979bdb4c00f08a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.908008 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fecdad6d120541bd8d3990cbd9b0a982. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.908555 1945 tablet_service.cc:1505] Processing CreateTablet for tablet c81d8aaeda0e437ca4e8640d5b7e1802 (DEFAULT_TABLE table=default.Table_53792bda07c940909d190b494a1e663b [id=690a41885744427d939902868684ff24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.908902 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81d8aaeda0e437ca4e8640d5b7e1802. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.909021 4067 raft_consensus.cc:493] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.909150 4067 raft_consensus.cc:515] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.909722 4067 leader_election.cc:290] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.910251 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a53f25688e3b45dca68984e2c543d094" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:48.910754 2194 leader_election.cc:343] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.911237 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 662a86b2a3af4ad4b5edee9e5d6ff90e (DEFAULT_TABLE table=default.Table_a08b6588e1414d00874ff1ec6c01c00b [id=8d288e42a17e4bf981c8b360a5d55c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.911598 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 662a86b2a3af4ad4b5edee9e5d6ff90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.912086 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a53f25688e3b45dca68984e2c543d094" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:48.912317 2116 raft_consensus.cc:2468] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:48.912464 2237 tablet_service.cc:1505] Processing CreateTablet for tablet c81d8aaeda0e437ca4e8640d5b7e1802 (DEFAULT_TABLE table=default.Table_53792bda07c940909d190b494a1e663b [id=690a41885744427d939902868684ff24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.912796 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81d8aaeda0e437ca4e8640d5b7e1802. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.913244 2098 tablet_service.cc:1505] Processing CreateTablet for tablet c81d8aaeda0e437ca4e8640d5b7e1802 (DEFAULT_TABLE table=default.Table_53792bda07c940909d190b494a1e663b [id=690a41885744427d939902868684ff24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.913618 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81d8aaeda0e437ca4e8640d5b7e1802. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.920078 4098 raft_consensus.cc:493] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.920261 4098 raft_consensus.cc:515] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.920850 4098 leader_election.cc:290] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.921645 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "885c10070d6942f7abe52c4558d887e6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:48.922441 2053 leader_election.cc:343] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.923154 2195 leader_election.cc:304] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.923462 4067 raft_consensus.cc:2804] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:48.923561 4067 raft_consensus.cc:493] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:48.923638 4067 raft_consensus.cc:3060] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.926332 4067 raft_consensus.cc:515] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.927150 4067 leader_election.cc:290] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.928422 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a53f25688e3b45dca68984e2c543d094" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:48.928470 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a53f25688e3b45dca68984e2c543d094" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:48.928643 2116 raft_consensus.cc:3060] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:48.929064 2194 leader_election.cc:343] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.929376 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "885c10070d6942f7abe52c4558d887e6" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:48.930634 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 662a86b2a3af4ad4b5edee9e5d6ff90e (DEFAULT_TABLE table=default.Table_a08b6588e1414d00874ff1ec6c01c00b [id=8d288e42a17e4bf981c8b360a5d55c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.931133 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 662a86b2a3af4ad4b5edee9e5d6ff90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.933259 2376 ts_tablet_manager.cc:1916] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:48.934782 2053 leader_election.cc:343] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:48.934913 2053 leader_election.cc:304] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:48.935403 4098 raft_consensus.cc:2749] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:48.940488 1656 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_bc60cafa54fd4cceb4f7efece8d873b5 [id=c8435cef755d4de69c33f599df1c0f75] alter complete (version 5)
I20251026 02:56:48.941020 1654 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_24c7322324534108a75d2fcae7d1082b [id=de5272bcc29644208ddd95bdec21254e] alter complete (version 2)
I20251026 02:56:48.941171 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet fe27c85771964e4ca0ea747bbb4826df
I20251026 02:56:48.941962 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e5f896de4e0d426da281ce2ec2724a04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:48.942623 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5f896de4e0d426da281ce2ec2724a04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.945976 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet fe27c85771964e4ca0ea747bbb4826df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.935984 1944 tablet_service.cc:1505] Processing CreateTablet for tablet fecdad6d120541bd8d3990cbd9b0a982 (DEFAULT_TABLE table=default.Table_7549c6a144774b478402b14065800d6b [id=6255d790c1f7481c979bdb4c00f08a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.948474 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet fe27c85771964e4ca0ea747bbb4826df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.948666 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fecdad6d120541bd8d3990cbd9b0a982. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.948808 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet fe27c85771964e4ca0ea747bbb4826df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.952033 2376 ts_tablet_manager.cc:1929] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:48.954485 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet cedff3287c7b4a07a81462e8c4b44970
I20251026 02:56:48.955245 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_169fb347542948c896eb9cd54f60b8ff" } new_table_name: "default.Table_63cd1c9a6d0c48f6941a2e72e3d81142" modify_external_catalogs: true
I20251026 02:56:48.956941 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4464817693df4358810261706b8b254b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:48.956984 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_355df3b4f4f94da898a80afb145a00d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:48.957548 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_355df3b4f4f94da898a80afb145a00d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:48.957612 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4464817693df4358810261706b8b254b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.958909 1663 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_c30cfa52b6524323bda478613d74178b [id=1fd74186ea10498599435dccd7330d25] alter complete (version 1)
I20251026 02:56:48.958992 2376 log.cc:1199] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/fe48472efa444bf580e4074c96a9de92
I20251026 02:56:48.959358 2376 ts_tablet_manager.cc:1950] T fe48472efa444bf580e4074c96a9de92 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:48.960171 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 12e10fd463cd43c7abb22ef428c0aa37
I20251026 02:56:48.960728 1659 catalog_manager.cc:6719] ReplaceTablet: tablet 12e10fd463cd43c7abb22ef428c0aa37 deleted and replaced by tablet 4880eb8bc1214e80a2baeac5fa497727
I20251026 02:56:48.960918 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 977) took 3272 ms. Trace:
I20251026 02:56:48.961429 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_f516b13fdea9459b84f650d89ddb6b21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:48.962106 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f516b13fdea9459b84f650d89ddb6b21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:48.963472 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8306f2ece961480d906df88f40b9740b
I20251026 02:56:48.963469 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 122c81787d5c43d28b1fb639e3542c76 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:48.966581 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 8306f2ece961480d906df88f40b9740b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.969655 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 12e10fd463cd43c7abb22ef428c0aa37 with delete_type TABLET_DATA_DELETED (replaced by tablet 4880eb8bc1214e80a2baeac5fa497727) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.971974 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b77bb6b4722449c9699b548a9ef6314 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.972442 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fb42b54f558499090a1d7cf257dd84c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.961015 2376 rpcz_store.cc:276] 1026 02:56:45.688816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.688881 (+ 65us) service_pool.cc:225] Handling call
1026 02:56:48.847340 (+3158459us) ts_tablet_manager.cc:1151] Deleting tablet fe48472efa444bf580e4074c96a9de92
1026 02:56:48.847351 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.948060 (+100709us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.951936 (+ 3876us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.960898 (+ 8962us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3158256}
I20251026 02:56:48.972613 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 41b7b0c36ba748f0a1565892dbe98a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.973156 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 978) took 3203 ms. Trace:
I20251026 02:56:48.973277 2376 rpcz_store.cc:276] 1026 02:56:45.769965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.770066 (+ 101us) service_pool.cc:225] Handling call
1026 02:56:48.973019 (+3202953us) ts_tablet_manager.cc:1151] Deleting tablet 53264ef08ff7454faa19b18fa35bd803
1026 02:56:48.973032 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.973135 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3202708}
I20251026 02:56:48.973644 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 979) took 3199 ms. Trace:
I20251026 02:56:48.973742 2376 rpcz_store.cc:276] 1026 02:56:45.773884 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.774618 (+ 734us) service_pool.cc:225] Handling call
1026 02:56:48.973537 (+3198919us) ts_tablet_manager.cc:1151] Deleting tablet 885c10070d6942f7abe52c4558d887e6
1026 02:56:48.973547 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.973628 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3198723}
I20251026 02:56:48.974013 2376 tablet_replica.cc:333] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:48.974253 2376 raft_consensus.cc:2243] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.974458 2376 raft_consensus.cc:2272] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:48.975093 2376 ts_tablet_manager.cc:1916] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:48.975746 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 99d318aa0d1847c58167bf0af9e75aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.975870 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 8306f2ece961480d906df88f40b9740b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.975931 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 12e10fd463cd43c7abb22ef428c0aa37 with delete_type TABLET_DATA_DELETED (replaced by tablet 4880eb8bc1214e80a2baeac5fa497727) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.976359 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet fe48472efa444bf580e4074c96a9de92 (table default.Table_fc676ef48d7e4c7bbb2fac99cdf17176 [id=7e3d19f1d75d4ebb94c220327ff8abcb]) successfully deleted
W20251026 02:56:48.976923 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 885c10070d6942f7abe52c4558d887e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 885c10070d6942f7abe52c4558d887e6 already in progress: creating tablet
I20251026 02:56:48.977347 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.977636 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 31b6e08597bd450eb1875201cf594a82 (table default.Table_436964d7d7d54f38a72eebf1d65d8d6e [id=a557708372bd4249969f80adf8641556]). Expected version 3 got 0
I20251026 02:56:48.978528 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet cedff3287c7b4a07a81462e8c4b44970 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.978760 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 12e10fd463cd43c7abb22ef428c0aa37 with delete_type TABLET_DATA_DELETED (replaced by tablet 4880eb8bc1214e80a2baeac5fa497727) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.978967 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b5e61afe90e4bc4a2897de21e0b3919 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b562b7b51f94b268c667650df199fbd) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:48.979084 2195 consensus_peers.cc:597] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d43b6f22dd4f4f9c9657f21514550312. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:48.979147 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet c2d6691bb4cf4a8292b0c082e54a91a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d55e3a643d744e6b2d4485543d4b626) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.979228 4067 raft_consensus.cc:493] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:48.979105 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 8306f2ece961480d906df88f40b9740b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.979405 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 2fd8f02126b24285bbfa575540ba0f09 (table default.Table_7539d36a088f4eafaa21d151ed9ea1cb [id=1950d7cb826b4a99b1ba8866e1d845f3]). Expected version 1 got 0
I20251026 02:56:48.979349 4067 raft_consensus.cc:515] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.979617 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet c966cae6886048a2b6ad3b8bd7007bcb (table default.Table_8dadea9fa7254668b84991c8d7cc4f52 [id=f7982d61f7d8411193899caaa357b61b]). Expected version 1 got 0
I20251026 02:56:48.979836 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 3aea01a248b345cf81d4d9862744eb78 (table default.Table_d68a91a3695240a39e5526b9cff6f6f2 [id=74d6732406a448e49911361bdc64417a]). Expected version 2 got 0
I20251026 02:56:48.980068 1655 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet db94738da3554fbda64c23fde3ef6c1b (table default.Table_f9afa2d0968b401e8fc615844c100b5e [id=928effe263004633a026f61a978a1e8a]). Expected version 2 got 0
I20251026 02:56:48.980108 4067 leader_election.cc:290] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:48.980526 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b5e61afe90e4bc4a2897de21e0b3919" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:48.980710 1981 raft_consensus.cc:2468] T 4b5e61afe90e4bc4a2897de21e0b3919 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:48.981103 2194 leader_election.cc:304] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:48.981457 4067 raft_consensus.cc:2804] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:48.981583 4067 raft_consensus.cc:493] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:48.981693 4067 raft_consensus.cc:3060] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:48.982424 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet c31d2704d1f14ddead69c794cc631ec1 with delete_type TABLET_DATA_DELETED (replaced by tablet de9124fe1c8945d9b9565cb732bef20e) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.982601 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 21b936fa4f6a4ff6af321971a30cccbe with delete_type TABLET_DATA_DELETED (replaced by tablet a956ebfb1eaa4d6fb1575b4165293193) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:48.984647 4067 raft_consensus.cc:515] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:48.984948 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ac38e6257e74367b06b15fc49d2af81 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c148c74fbcd4d30b9ba0ea059284427) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:48.987303 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b5e61afe90e4bc4a2897de21e0b3919" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:48.991247 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fb2ff87cf0248a0a4a77f2733ef867b with delete_type TABLET_DATA_DELETED (replaced by tablet ce11081bf85f44e8bf1f92ce0f4bf152) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.991529 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cf76ef8f6af445eb0ce15c8d79136c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:48.994308 2098 tablet_service.cc:1505] Processing CreateTablet for tablet f9302b7d4cb64f0594ad8dc41d376d3b (DEFAULT_TABLE table=default.Table_acb7127a0c1946b79991b8e0f61352bf [id=c67fd697cf6e4d33a7a73e82c2cc75a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.994386 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 4f8514198b7e4d4380d621767b93b59b (DEFAULT_TABLE table=default.Table_9b0b596ec6014ba288a0cb8110dfc2a2 [id=c42fcf63bd4b4f6091a6291aa5c812e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.994786 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f8514198b7e4d4380d621767b93b59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.994982 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 2188765cd2d54d49a0edab1453e0e52a (DEFAULT_TABLE table=default.Table_50ea87134e5c4bba934f9f7a5e275c9b [id=34a9fbe2da4d47e9a600083ae1a85971]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:48.995319 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2188765cd2d54d49a0edab1453e0e52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:48.997854 2372 ts_tablet_manager.cc:1929] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:48.997959 2372 log.cc:1199] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/4b5e61afe90e4bc4a2897de21e0b3919
I20251026 02:56:48.998265 2372 ts_tablet_manager.cc:1950] T 4b5e61afe90e4bc4a2897de21e0b3919 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:48.998994 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 981) took 5086 ms. Trace:
I20251026 02:56:48.999073 2372 rpcz_store.cc:276] 1026 02:56:43.912858 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.912906 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:48.743728 (+4830822us) ts_tablet_manager.cc:1151] Deleting tablet 4b5e61afe90e4bc4a2897de21e0b3919
1026 02:56:48.743742 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:48.771081 (+ 27339us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.997753 (+226672us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:48.998974 (+ 1221us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4827556}
I20251026 02:56:48.999377 2372 tablet_replica.cc:333] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:48.999552 2372 raft_consensus.cc:2243] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:48.999676 2372 raft_consensus.cc:2272] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:49.000133 2372 ts_tablet_manager.cc:1916] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:49.000823 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 4b5e61afe90e4bc4a2897de21e0b3919 (table default.Table_30b96d2d4369458390803c6be18efe0e [id=9fd6e480fc3543149959773e691a3ddd]) successfully deleted
I20251026 02:56:49.003909 1656 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 7abcdac4a70b48ad9fbba1bc8a60283b (table default.Table_30611e161c274e8b99bd1251e5e527aa [id=b5d91d3e8a2643c09fee85976972391a]). Expected version 1 got 0
I20251026 02:56:49.004443 1656 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 386e1bd4635b4a7e8e7f249b577fd978 (table default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2]). Expected version 2 got 0
I20251026 02:56:49.005031 1954 tablet_service.cc:1505] Processing CreateTablet for tablet f9302b7d4cb64f0594ad8dc41d376d3b (DEFAULT_TABLE table=default.Table_acb7127a0c1946b79991b8e0f61352bf [id=c67fd697cf6e4d33a7a73e82c2cc75a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.005193 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 4f8514198b7e4d4380d621767b93b59b (DEFAULT_TABLE table=default.Table_9b0b596ec6014ba288a0cb8110dfc2a2 [id=c42fcf63bd4b4f6091a6291aa5c812e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.005580 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f8514198b7e4d4380d621767b93b59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.005746 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 2188765cd2d54d49a0edab1453e0e52a (DEFAULT_TABLE table=default.Table_50ea87134e5c4bba934f9f7a5e275c9b [id=34a9fbe2da4d47e9a600083ae1a85971]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.006085 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2188765cd2d54d49a0edab1453e0e52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.009529 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9302b7d4cb64f0594ad8dc41d376d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.004940 1656 catalog_manager.cc:5649] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:49.013849 1656 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 41a1442fffe54add93666e3c78a0cc58 (table default.Table_1faa0c8151724c5e816546ee2e1d5182 [id=a106f9ece0b9467a8bc1d5d4b3524c41]). Expected version 1 got 0
I20251026 02:56:49.017041 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet cedff3287c7b4a07a81462e8c4b44970 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.017490 4167 raft_consensus.cc:493] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.017649 4167 raft_consensus.cc:515] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.018301 4167 leader_election.cc:290] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.018647 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 95c9fbae62db4850b0f02a2443a3432b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.018837 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 41b7b0c36ba748f0a1565892dbe98a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:49.019008 1916 leader_election.cc:343] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 41b7b0c36ba748f0a1565892dbe98a78
I20251026 02:56:49.019143 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f22c6c163c34ed48a0b7eeb5a574e76 with delete_type TABLET_DATA_DELETED (replaced by tablet e4c149e8b7f9420c8dba56da2e21f3f9) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.019006 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a345fa5edbd48d2891b8b40b09cb3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.019057 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet c2d6691bb4cf4a8292b0c082e54a91a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d55e3a643d744e6b2d4485543d4b626) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.019431 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_deeb6a15a5424e609e622a6c71b3553a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.020049 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_deeb6a15a5424e609e622a6c71b3553a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.021107 1946 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a3f3fc3c7a04d05bae28472251f3acc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.021531 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_30b96d2d4369458390803c6be18efe0e" } new_table_name: "default.Table_ca64283c7621400ba743ec5cff194692" modify_external_catalogs: true
I20251026 02:56:49.021708 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9302b7d4cb64f0594ad8dc41d376d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.021886 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet cedff3287c7b4a07a81462e8c4b44970 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:48 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.022055 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b5e61afe90e4bc4a2897de21e0b3919 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b562b7b51f94b268c667650df199fbd) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.022657 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 81775055d98349f696cdab6171ffd58b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.025099 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet b8fe7825fa8d46669772cf84bc4b3afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.025520 2376 ts_tablet_manager.cc:1929] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:49.025637 2376 log.cc:1199] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/aa10cd990a944eab8d9eb66aaf20080c
I20251026 02:56:49.025946 2376 ts_tablet_manager.cc:1950] T aa10cd990a944eab8d9eb66aaf20080c P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:49.026332 4167 raft_consensus.cc:493] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.026449 4167 raft_consensus.cc:515] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.027015 4167 leader_election.cc:290] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:48.988461 2377 ts_tablet_manager.cc:1916] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:49.027715 1916 leader_election.cc:343] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: c2d6691bb4cf4a8292b0c082e54a91a5
I20251026 02:56:49.028167 4067 leader_election.cc:290] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:49.028254 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 24f16e40a30c41788a6f9813c235c5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.028868 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet a3af2695c41a4439bc4092b7cb1b2edf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.028846 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b5e61afe90e4bc4a2897de21e0b3919" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:49.028997 1981 raft_consensus.cc:3060] T 4b5e61afe90e4bc4a2897de21e0b3919 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.031695 1981 raft_consensus.cc:2468] T 4b5e61afe90e4bc4a2897de21e0b3919 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:49.032058 2194 leader_election.cc:304] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:49.034752 2377 ts_tablet_manager.cc:1929] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:49.034909 2377 log.cc:1199] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/ec13c5386abe414ba25589e03d5b850a
I20251026 02:56:49.035333 2377 ts_tablet_manager.cc:1950] T ec13c5386abe414ba25589e03d5b850a P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:49.035645 4067 raft_consensus.cc:2804] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 02:56:48.989411 2195 leader_election.cc:343] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: Tablet not RUNNING: STOPPED
I20251026 02:56:48.995402 2381 tablet_bootstrap.cc:492] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:49.035890 4067 raft_consensus.cc:697] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:49.035926 2381 ts_tablet_manager.cc:1403] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.273s user 0.033s sys 0.010s
I20251026 02:56:49.036532 4067 consensus_queue.cc:237] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.036666 2381 raft_consensus.cc:359] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.036782 2381 raft_consensus.cc:385] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.036825 2381 raft_consensus.cc:740] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.036980 2381 consensus_queue.cc:260] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.037062 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 892a5fab2af1432eb31d92d9859ea2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.037452 2381 ts_tablet_manager.cc:1434] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 02:56:49.037688 2381 tablet_bootstrap.cc:492] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:49.038237 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 950) took 5866 ms. Trace:
I20251026 02:56:49.038457 2375 tablet_bootstrap.cc:492] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:49.038323 2377 rpcz_store.cc:276] 1026 02:56:43.171802 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.173426 (+ 1624us) service_pool.cc:225] Handling call
1026 02:56:48.766811 (+5593385us) ts_tablet_manager.cc:1151] Deleting tablet ec13c5386abe414ba25589e03d5b850a
1026 02:56:48.766824 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.030718 (+263894us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.034642 (+ 3924us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.038217 (+ 3575us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2077,"tablet-delete.queue_time_us":5593256}
I20251026 02:56:49.038581 2375 ts_tablet_manager.cc:1403] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.217s user 0.013s sys 0.007s
I20251026 02:56:49.038789 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 961) took 5593 ms. Trace:
I20251026 02:56:49.038853 2377 rpcz_store.cc:276] 1026 02:56:43.445662 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.445776 (+ 114us) service_pool.cc:225] Handling call
1026 02:56:49.038663 (+5592887us) ts_tablet_manager.cc:1151] Deleting tablet 868dbbfce40c4dd6b36cc220966b1986
1026 02:56:49.038673 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.038775 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5592656}
I20251026 02:56:49.039124 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 962) took 5593 ms. Trace:
I20251026 02:56:49.039285 2381 tablet_bootstrap.cc:654] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.039197 2377 rpcz_store.cc:276] 1026 02:56:43.445738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.446048 (+ 310us) service_pool.cc:225] Handling call
1026 02:56:49.039031 (+5592983us) ts_tablet_manager.cc:1151] Deleting tablet 9b9dc0825e0844f68d5afcc201db737c
1026 02:56:49.039039 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.039110 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5592852}
I20251026 02:56:49.039429 2375 raft_consensus.cc:359] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.039523 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 954) took 5594 ms. Trace:
I20251026 02:56:49.039601 2375 raft_consensus.cc:385] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.039664 2375 raft_consensus.cc:740] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.039636 2377 rpcz_store.cc:276] 1026 02:56:43.445137 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.448254 (+ 3117us) service_pool.cc:225] Handling call
1026 02:56:49.039405 (+5591151us) ts_tablet_manager.cc:1151] Deleting tablet 19918cb3b7d0450f9de17026f8ea3295
1026 02:56:49.039412 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.039510 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5591014}
I20251026 02:56:49.039938 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 965) took 5594 ms. Trace:
I20251026 02:56:49.039875 2375 consensus_queue.cc:260] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.040001 2377 rpcz_store.cc:276] 1026 02:56:43.445952 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.452327 (+ 6375us) service_pool.cc:225] Handling call
1026 02:56:49.039825 (+5587498us) ts_tablet_manager.cc:1151] Deleting tablet 035724949fe142ea93cb998f8fdc4344
1026 02:56:49.039835 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.039924 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5587290}
W20251026 02:56:49.040781 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 51378.8us
I20251026 02:56:49.044690 2375 ts_tablet_manager.cc:1434] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251026 02:56:49.045339 2375 tablet_bootstrap.cc:492] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:49.046061 2081 tablet_service.cc:1505] Processing CreateTablet for tablet f9bf135f90e54521a3ee33b7695ed424 (DEFAULT_TABLE table=default.Table_658f91f2bd254d1285c47121611a34af [id=6ebec9b03b9c4a1bb87c0ee68eadef20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.046448 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9bf135f90e54521a3ee33b7695ed424. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:49.046494 2194 consensus_peers.cc:597] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:49.047204 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41b7b0c36ba748f0a1565892dbe98a78" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.048074 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2d6691bb4cf4a8292b0c082e54a91a5" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.048257 2259 raft_consensus.cc:2468] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:49.049024 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 1e3df65c6ab04995af195af5893438c5 (DEFAULT_TABLE table=default.Table_066e72649e794e878a6512b6dea2f868 [id=bad03d36ada148d88058d99d20c45072]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.049054 2092 tablet_service.cc:1505] Processing CreateTablet for tablet a0b9626c8ed14636acdf0a286a709625 (DEFAULT_TABLE table=default.Table_73be634ceddf488ea518d690587f6cd0 [id=5e42165433a248a9bff99b74846d72e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.049407 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3df65c6ab04995af195af5893438c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.049582 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0b9626c8ed14636acdf0a286a709625. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:49.049772 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92ae57c9a65a495e99e61efa123e8e88 consensus state has no leader
I20251026 02:56:49.049953 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 6726bb05f01141c6bd12c27497da5cf7 (DEFAULT_TABLE table=default.Table_d04da838fd9f42feab6cb9d6178160a3 [id=c0c2db1fb85e44219fa0eede60df665a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.050158 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_42b54262967149fda8be1e83c4579b5d" } new_table_name: "default.Table_2060154e868f41c3891ac2af97a423cc" modify_external_catalogs: true
I20251026 02:56:49.051924 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7e72edb3045a47d1ab44c37eadf3c705"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.052511 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e72edb3045a47d1ab44c37eadf3c705 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.054654 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6726bb05f01141c6bd12c27497da5cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.056473 1943 tablet_service.cc:1505] Processing CreateTablet for tablet a0b9626c8ed14636acdf0a286a709625 (DEFAULT_TABLE table=default.Table_73be634ceddf488ea518d690587f6cd0 [id=5e42165433a248a9bff99b74846d72e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.056840 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0b9626c8ed14636acdf0a286a709625. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.056855 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet ec13c5386abe414ba25589e03d5b850a (table default.Table_3792bcbbc16b4be0b113cb5f07b9261b [id=8a3d1c20763445c499bd655eb99deb56]) successfully deleted
W20251026 02:56:49.057080 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 868dbbfce40c4dd6b36cc220966b1986 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 868dbbfce40c4dd6b36cc220966b1986 already in progress: creating tablet
W20251026 02:56:49.057274 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 9b9dc0825e0844f68d5afcc201db737c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b9dc0825e0844f68d5afcc201db737c already in progress: creating tablet
W20251026 02:56:49.057427 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 19918cb3b7d0450f9de17026f8ea3295 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19918cb3b7d0450f9de17026f8ea3295 already in progress: creating tablet
W20251026 02:56:49.057595 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 035724949fe142ea93cb998f8fdc4344 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 035724949fe142ea93cb998f8fdc4344 already in progress: creating tablet
I20251026 02:56:49.058840 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 4f8514198b7e4d4380d621767b93b59b (DEFAULT_TABLE table=default.Table_9b0b596ec6014ba288a0cb8110dfc2a2 [id=c42fcf63bd4b4f6091a6291aa5c812e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.059219 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f8514198b7e4d4380d621767b93b59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.059636 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 960) took 5614 ms. Trace:
I20251026 02:56:49.059713 2377 rpcz_store.cc:276] 1026 02:56:43.445591 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.454340 (+ 8749us) service_pool.cc:225] Handling call
1026 02:56:49.040201 (+5585861us) ts_tablet_manager.cc:1151] Deleting tablet 0245af5a2dcb4f569b62307965895c1f
1026 02:56:49.040211 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.059615 (+ 19404us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5585715}
I20251026 02:56:49.060050 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 970) took 5507 ms. Trace:
I20251026 02:56:49.060107 2377 rpcz_store.cc:276] 1026 02:56:43.552538 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.552595 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:49.059953 (+5507358us) ts_tablet_manager.cc:1151] Deleting tablet bcd5621aa3754733b22508be89f080db
1026 02:56:49.059962 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.060038 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5507184}
I20251026 02:56:49.060355 2377 tablet_replica.cc:333] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:49.060434 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 6726bb05f01141c6bd12c27497da5cf7 (DEFAULT_TABLE table=default.Table_d04da838fd9f42feab6cb9d6178160a3 [id=c0c2db1fb85e44219fa0eede60df665a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.060560 2377 raft_consensus.cc:2243] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:49.060747 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6726bb05f01141c6bd12c27497da5cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.060865 2377 raft_consensus.cc:2272] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:49.056133 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 1e3df65c6ab04995af195af5893438c5 (DEFAULT_TABLE table=default.Table_066e72649e794e878a6512b6dea2f868 [id=bad03d36ada148d88058d99d20c45072]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.061425 2377 ts_tablet_manager.cc:1916] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:49.061621 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3df65c6ab04995af195af5893438c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.065799 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 2188765cd2d54d49a0edab1453e0e52a (DEFAULT_TABLE table=default.Table_50ea87134e5c4bba934f9f7a5e275c9b [id=34a9fbe2da4d47e9a600083ae1a85971]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.067871 2378 tablet_bootstrap.cc:654] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.070518 2372 ts_tablet_manager.cc:1929] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:49.070648 2372 log.cc:1199] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/f2d97084b5914350afbfda5c2a34bf9a
I20251026 02:56:49.071549 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 4e8e6ce9b8cf479c87b3d818572f2828 (DEFAULT_TABLE table=default.Table_759cf0037bcd4bf58341bd2f01711cdb [id=7dcf1c9999e741d485fab1bf38a8409f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.072638 2237 tablet_service.cc:1505] Processing CreateTablet for tablet f9bf135f90e54521a3ee33b7695ed424 (DEFAULT_TABLE table=default.Table_658f91f2bd254d1285c47121611a34af [id=6ebec9b03b9c4a1bb87c0ee68eadef20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.072820 1915 leader_election.cc:304] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:49.073581 4167 raft_consensus.cc:2749] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.073846 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 6726bb05f01141c6bd12c27497da5cf7 (DEFAULT_TABLE table=default.Table_d04da838fd9f42feab6cb9d6178160a3 [id=c0c2db1fb85e44219fa0eede60df665a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.074227 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 1e3df65c6ab04995af195af5893438c5 (DEFAULT_TABLE table=default.Table_066e72649e794e878a6512b6dea2f868 [id=bad03d36ada148d88058d99d20c45072]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.074339 2375 tablet_bootstrap.cc:654] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.080781 1915 leader_election.cc:304] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:49.081302 4167 raft_consensus.cc:2804] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:49.081405 4167 raft_consensus.cc:493] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:49.081467 4167 raft_consensus.cc:3060] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.084592 1945 tablet_service.cc:1505] Processing CreateTablet for tablet f9bf135f90e54521a3ee33b7695ed424 (DEFAULT_TABLE table=default.Table_658f91f2bd254d1285c47121611a34af [id=6ebec9b03b9c4a1bb87c0ee68eadef20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.084107 4167 raft_consensus.cc:515] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.085126 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 4e8e6ce9b8cf479c87b3d818572f2828 (DEFAULT_TABLE table=default.Table_759cf0037bcd4bf58341bd2f01711cdb [id=7dcf1c9999e741d485fab1bf38a8409f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.085536 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9bf135f90e54521a3ee33b7695ed424. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.085911 4167 leader_election.cc:290] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:49.086491 1916 leader_election.cc:343] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: c2d6691bb4cf4a8292b0c082e54a91a5
I20251026 02:56:49.086581 2224 tablet_service.cc:1505] Processing CreateTablet for tablet f9302b7d4cb64f0594ad8dc41d376d3b (DEFAULT_TABLE table=default.Table_acb7127a0c1946b79991b8e0f61352bf [id=c67fd697cf6e4d33a7a73e82c2cc75a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.088152 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 4e8e6ce9b8cf479c87b3d818572f2828 (DEFAULT_TABLE table=default.Table_759cf0037bcd4bf58341bd2f01711cdb [id=7dcf1c9999e741d485fab1bf38a8409f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:49.087486 2195 leader_election.cc:343] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 4b5e61afe90e4bc4a2897de21e0b3919
I20251026 02:56:49.090678 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 980) took 3310 ms. Trace:
I20251026 02:56:49.090757 2376 rpcz_store.cc:276] 1026 02:56:45.780407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.781114 (+ 707us) service_pool.cc:225] Handling call
1026 02:56:48.973966 (+3192852us) ts_tablet_manager.cc:1151] Deleting tablet aa10cd990a944eab8d9eb66aaf20080c
1026 02:56:48.973974 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.005506 (+ 31532us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.025427 (+ 19921us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.090656 (+ 65229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3192647}
I20251026 02:56:49.091656 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2d6691bb4cf4a8292b0c082e54a91a5" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:49.091864 2259 raft_consensus.cc:3060] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.094713 2259 raft_consensus.cc:2468] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:49.095283 1915 leader_election.cc:304] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:49.096036 4167 raft_consensus.cc:2804] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:49.096274 4167 raft_consensus.cc:697] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:49.096819 4167 consensus_queue.cc:237] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:49.097039 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0245af5a2dcb4f569b62307965895c1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0245af5a2dcb4f569b62307965895c1f already in progress: creating tablet
I20251026 02:56:49.096150 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 981) took 3157 ms. Trace:
I20251026 02:56:49.100636 2376 rpcz_store.cc:276] 1026 02:56:45.938971 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.939041 (+ 70us) service_pool.cc:225] Handling call
1026 02:56:49.096027 (+3156986us) ts_tablet_manager.cc:1151] Deleting tablet 9b976c87b5d04f9294add7761a2afbfd
1026 02:56:49.096036 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.096131 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3156758}
I20251026 02:56:49.101774 2375 tablet_bootstrap.cc:492] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:49.101922 2375 ts_tablet_manager.cc:1403] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20251026 02:56:49.102620 2375 raft_consensus.cc:359] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.102828 2375 raft_consensus.cc:385] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.102892 2375 raft_consensus.cc:740] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.103435 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 988) took 2944 ms. Trace:
I20251026 02:56:49.103526 2376 rpcz_store.cc:276] 1026 02:56:46.159091 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.159154 (+ 63us) service_pool.cc:225] Handling call
1026 02:56:49.103306 (+2944152us) ts_tablet_manager.cc:1151] Deleting tablet 075f6ca8787943a09443863c544a1284
1026 02:56:49.103316 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.103415 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2943939}
I20251026 02:56:49.104341 2381 tablet_bootstrap.cc:492] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:49.104636 2381 ts_tablet_manager.cc:1403] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.067s user 0.005s sys 0.002s
I20251026 02:56:49.105444 2381 raft_consensus.cc:359] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.105680 2381 raft_consensus.cc:385] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.105769 2381 raft_consensus.cc:740] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.097565 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fb45779e61d94a69ab237d499fdaea52" } new_table_name: "default.Table_e378b0f02fc1489886686e5f52d05d40" modify_external_catalogs: true
I20251026 02:56:49.106029 2381 consensus_queue.cc:260] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:49.105897 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet bcd5621aa3754733b22508be89f080db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcd5621aa3754733b22508be89f080db already in progress: creating tablet
I20251026 02:56:49.106628 2381 ts_tablet_manager.cc:1434] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:49.113013 2381 tablet_bootstrap.cc:492] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:49.115166 2381 tablet_bootstrap.cc:654] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.116904 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 989) took 2957 ms. Trace:
I20251026 02:56:49.117091 2376 rpcz_store.cc:276] 1026 02:56:46.159223 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.159367 (+ 144us) service_pool.cc:225] Handling call
1026 02:56:49.116742 (+2957375us) ts_tablet_manager.cc:1151] Deleting tablet 142043b5d558490c9399626c3764ff7e
1026 02:56:49.116757 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.116880 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2957199}
I20251026 02:56:49.117650 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 990) took 2935 ms. Trace:
I20251026 02:56:49.117779 2376 rpcz_store.cc:276] 1026 02:56:46.182115 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.182178 (+ 63us) service_pool.cc:225] Handling call
1026 02:56:49.117546 (+2935368us) ts_tablet_manager.cc:1151] Deleting tablet 586bd6e40a7a4785b71b2a3d542e2411
1026 02:56:49.117555 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.117635 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2935203}
I20251026 02:56:49.118264 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 997) took 2860 ms. Trace:
I20251026 02:56:49.118383 2376 rpcz_store.cc:276] 1026 02:56:46.257790 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.257839 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:49.118166 (+2860327us) ts_tablet_manager.cc:1151] Deleting tablet e4c149e8b7f9420c8dba56da2e21f3f9
1026 02:56:49.118175 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.118251 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2860173}
I20251026 02:56:49.118825 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 998) took 2860 ms. Trace:
I20251026 02:56:49.118948 2376 rpcz_store.cc:276] 1026 02:56:46.257888 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.258045 (+ 157us) service_pool.cc:225] Handling call
1026 02:56:49.118744 (+2860699us) ts_tablet_manager.cc:1151] Deleting tablet 7bef60e3a10843b4be2022ab4b64be68
1026 02:56:49.118752 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.118813 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2860575}
I20251026 02:56:49.119338 2376 tablet_replica.cc:333] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:49.119553 2376 raft_consensus.cc:2243] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:49.119758 2376 raft_consensus.cc:2272] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:49.119943 4068 raft_consensus.cc:493] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.120075 4068 raft_consensus.cc:515] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.120740 4068 leader_election.cc:290] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:49.121757 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d97084b5914350afbfda5c2a34bf9a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.121904 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d97084b5914350afbfda5c2a34bf9a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:49.105006 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e8e6ce9b8cf479c87b3d818572f2828. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:49.123140 2194 leader_election.cc:343] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:49.125501 2195 leader_election.cc:343] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: Tablet not RUNNING: STOPPED
I20251026 02:56:49.125689 2195 leader_election.cc:304] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.125954 4068 raft_consensus.cc:2749] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.127831 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet aa10cd990a944eab8d9eb66aaf20080c (table default.Table_42c4c5e055054536b2b541711288a481 [id=503128cdfe654cfb9713c8e241eac740]) successfully deleted
W20251026 02:56:49.128165 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 9b976c87b5d04f9294add7761a2afbfd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b976c87b5d04f9294add7761a2afbfd already in progress: creating tablet
W20251026 02:56:49.128613 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 075f6ca8787943a09443863c544a1284 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 075f6ca8787943a09443863c544a1284 already in progress: creating tablet
W20251026 02:56:49.128903 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 142043b5d558490c9399626c3764ff7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 142043b5d558490c9399626c3764ff7e already in progress: creating tablet
W20251026 02:56:49.129153 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 586bd6e40a7a4785b71b2a3d542e2411 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 586bd6e40a7a4785b71b2a3d542e2411 already in progress: creating tablet
W20251026 02:56:49.129516 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 7bef60e3a10843b4be2022ab4b64be68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bef60e3a10843b4be2022ab4b64be68 already in progress: creating tablet
W20251026 02:56:49.132750 3337 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.1.150.61:37655 timed out after 0.715s (SENT)
I20251026 02:56:49.133460 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e8e6ce9b8cf479c87b3d818572f2828. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.144377 2375 consensus_queue.cc:260] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.144958 2375 ts_tablet_manager.cc:1434] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.043s user 0.001s sys 0.000s
I20251026 02:56:49.145227 2372 ts_tablet_manager.cc:1950] T f2d97084b5914350afbfda5c2a34bf9a P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:49.145644 1660 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 24449cb9121642009dec41220e27b193 (table default.Table_12cdd920c7d54ce7818f9fa6d5fce036 [id=e8ca26900f06405d904f48d7387baf16]). Expected version 1 got 0
I20251026 02:56:49.146370 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d9f69f33702146e281d6d9bfa43e968b" } modify_external_catalogs: true
I20251026 02:56:49.146579 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d9f69f33702146e281d6d9bfa43e968b" } modify_external_catalogs: true
I20251026 02:56:49.148142 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5068032512e84748bf5f2399a08bfa05" } modify_external_catalogs: true
I20251026 02:56:49.148326 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5068032512e84748bf5f2399a08bfa05" } modify_external_catalogs: true
I20251026 02:56:49.149466 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 122c81787d5c43d28b1fb639e3542c76
I20251026 02:56:49.149641 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_272aa94f33a74e0f99feb2cb36453493"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:49.149904 1654 catalog_manager.cc:6719] ReplaceTablet: tablet 122c81787d5c43d28b1fb639e3542c76 deleted and replaced by tablet b0b27d526b83492cbfa22604c943f76c
W20251026 02:56:49.150204 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_272aa94f33a74e0f99feb2cb36453493 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.151352 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_37167b3e100a458d81c0578a964bee27" } new_table_name: "default.Table_b045d6ff6d3b43d78d2a19383b8cd350" modify_external_catalogs: true
I20251026 02:56:49.154572 2376 ts_tablet_manager.cc:1916] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:49.155437 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b562b7b51f94b268c667650df199fbd consensus state has no leader
I20251026 02:56:49.158993 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 9b3bb056c94149bbb9f3dbfeb6db0d5a (DEFAULT_TABLE table=default.Table_debeb38e035a435fa936ab479b12d30e [id=caf30a612cba4ce6954ab61eacc8ba00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.159418 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b3bb056c94149bbb9f3dbfeb6db0d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.160470 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 982) took 5247 ms. Trace:
I20251026 02:56:49.160569 2372 rpcz_store.cc:276] 1026 02:56:43.913350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.913398 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:48.999334 (+5085936us) ts_tablet_manager.cc:1151] Deleting tablet f2d97084b5914350afbfda5c2a34bf9a
1026 02:56:48.999343 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.067005 (+ 67662us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.070403 (+ 3398us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.160447 (+ 90044us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5082609}
I20251026 02:56:49.160910 2372 tablet_replica.cc:333] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:49.160874 1654 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 2fd8f02126b24285bbfa575540ba0f09 (table default.Table_7539d36a088f4eafaa21d151ed9ea1cb [id=1950d7cb826b4a99b1ba8866e1d845f3]). Expected version 1 got 0
I20251026 02:56:49.161070 2372 raft_consensus.cc:2243] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:49.161233 2372 raft_consensus.cc:2272] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:49.161815 2372 ts_tablet_manager.cc:1916] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:49.163923 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 398c1f420974453b9b6b0fe17795378b (table default.Table_a92e68d05f4b455d810180ee4cc5874d [id=7f64fbec3582415a8924d01e14856a56]). Expected version 1 got 0
I20251026 02:56:49.164885 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet f2d97084b5914350afbfda5c2a34bf9a (table default.Table_a7b1603151c54757bd2505c7958e3bae [id=cd7cfcd0eac044609fe5b2751ac6a98a]) successfully deleted
I20251026 02:56:49.165228 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 79c18ebf56e045b6a52fae8871a9107f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.165557 4167 consensus_queue.cc:1048] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:49.166498 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 122c81787d5c43d28b1fb639e3542c76 with delete_type TABLET_DATA_DELETED (replaced by tablet b0b27d526b83492cbfa22604c943f76c) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.169099 4067 raft_consensus.cc:493] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.169375 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9bf135f90e54521a3ee33b7695ed424. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.169565 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3df65c6ab04995af195af5893438c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.169754 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e8e6ce9b8cf479c87b3d818572f2828. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.169994 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6726bb05f01141c6bd12c27497da5cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.172896 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2188765cd2d54d49a0edab1453e0e52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.173731 2116 raft_consensus.cc:2468] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:49.182188 2375 tablet_bootstrap.cc:492] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:49.184089 2375 tablet_bootstrap.cc:654] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.169565 4067 raft_consensus.cc:515] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.186100 4067 leader_election.cc:290] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:49.187163 2381 tablet_bootstrap.cc:492] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:49.187345 2381 ts_tablet_manager.cc:1403] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.002s
I20251026 02:56:49.188073 2381 raft_consensus.cc:359] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.188325 2381 raft_consensus.cc:385] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.188446 2381 raft_consensus.cc:740] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.188433 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b9a5daa36b547a7888d38dc3329eaf2" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:49.188951 2194 leader_election.cc:343] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.188689 2381 consensus_queue.cc:260] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.189673 2381 ts_tablet_manager.cc:1434] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 02:56:49.189843 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9302b7d4cb64f0594ad8dc41d376d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.190130 2381 tablet_bootstrap.cc:492] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:49.193962 2195 leader_election.cc:304] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.194857 4068 raft_consensus.cc:2804] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:49.194968 4068 raft_consensus.cc:697] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:49.195187 4068 consensus_queue.cc:237] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.202245 2378 tablet_bootstrap.cc:492] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:49.203104 2378 ts_tablet_manager.cc:1403] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.384s user 0.004s sys 0.010s
I20251026 02:56:49.204079 2378 raft_consensus.cc:359] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.204838 3997 raft_consensus.cc:493] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.205014 3997 raft_consensus.cc:515] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.205793 3997 leader_election.cc:290] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:49.206192 2195 leader_election.cc:343] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9b9a5daa36b547a7888d38dc3329eaf2
I20251026 02:56:49.206300 2195 leader_election.cc:304] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:49.207437 2194 consensus_peers.cc:597] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa10cd990a944eab8d9eb66aaf20080c. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:49.207342 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e203fe0403594d30b44d531153d42a10" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.208141 2378 raft_consensus.cc:385] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251026 02:56:49.208267 2053 leader_election.cc:343] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.208456 2378 raft_consensus.cc:740] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.208868 2378 consensus_queue.cc:260] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:49.209689 3337 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.61:37655, user_credentials={real_user=slave}} blocked reactor thread for 76991.8us
I20251026 02:56:49.212150 2378 ts_tablet_manager.cc:1434] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251026 02:56:49.215454 2377 ts_tablet_manager.cc:1929] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:49.216352 2377 log.cc:1199] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/f437d8b5f5784e8fbec10d0700e90348
I20251026 02:56:49.216764 2378 tablet_bootstrap.cc:492] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:49.218075 4167 raft_consensus.cc:493] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.221298 4167 raft_consensus.cc:515] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.222119 4167 leader_election.cc:290] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.222411 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0bacd15b41149b5856d20068711b432" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:49.223955 2378 tablet_bootstrap.cc:654] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.241560 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 122c81787d5c43d28b1fb639e3542c76 with delete_type TABLET_DATA_DELETED (replaced by tablet b0b27d526b83492cbfa22604c943f76c) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.243695 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 9b3bb056c94149bbb9f3dbfeb6db0d5a (DEFAULT_TABLE table=default.Table_debeb38e035a435fa936ab479b12d30e [id=caf30a612cba4ce6954ab61eacc8ba00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.246579 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b3bb056c94149bbb9f3dbfeb6db0d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.249521 2375 tablet_bootstrap.cc:492] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:49.249991 2375 ts_tablet_manager.cc:1403] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.001s
I20251026 02:56:49.250761 2375 raft_consensus.cc:359] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.250929 2375 raft_consensus.cc:385] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.250981 2375 raft_consensus.cc:740] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.251190 2375 consensus_queue.cc:260] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.251734 2375 ts_tablet_manager.cc:1434] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 02:56:49.252020 2375 tablet_bootstrap.cc:492] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:49.270331 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5b3d81865782468ca8ab3c0023b31352" } modify_external_catalogs: true
I20251026 02:56:49.270604 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5b3d81865782468ca8ab3c0023b31352" } modify_external_catalogs: true
I20251026 02:56:49.271384 3997 raft_consensus.cc:493] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.271536 3997 raft_consensus.cc:515] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.272109 3997 leader_election.cc:290] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.272425 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d4fb1ce9057f4cde849a4a8050bf6a18" } modify_external_catalogs: true
I20251026 02:56:49.272614 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d4fb1ce9057f4cde849a4a8050bf6a18" } modify_external_catalogs: true
I20251026 02:56:49.274832 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81775055d98349f696cdab6171ffd58b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:49.275398 2053 leader_election.cc:343] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.275787 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet d43b6f22dd4f4f9c9657f21514550312 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:27 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.287621 2372 ts_tablet_manager.cc:1929] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:49.287770 2372 log.cc:1199] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/e89a98fcfdbb4011876b0ebc2af91f67
I20251026 02:56:49.288098 2372 ts_tablet_manager.cc:1950] T e89a98fcfdbb4011876b0ebc2af91f67 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:49.294274 3997 raft_consensus.cc:493] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.294432 3997 raft_consensus.cc:515] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:49.294636 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7834a85168364390a841fc883681c845 consensus state has no leader
I20251026 02:56:49.295069 3997 leader_election.cc:290] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.295764 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85420a102dae4269811dbdc64aebb20a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:49.296248 2053 leader_election.cc:343] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.302806 2376 ts_tablet_manager.cc:1929] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:49.303014 2376 log.cc:1199] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/ec13c5386abe414ba25589e03d5b850a
I20251026 02:56:49.303464 2376 ts_tablet_manager.cc:1950] T ec13c5386abe414ba25589e03d5b850a P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:49.303820 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 999) took 2959 ms. Trace:
I20251026 02:56:49.241232 4069 raft_consensus.cc:2749] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.305341 2375 tablet_bootstrap.cc:654] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.253638 2381 tablet_bootstrap.cc:654] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.255118 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0bacd15b41149b5856d20068711b432" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.255610 2224 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b9a5daa36b547a7888d38dc3329eaf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d06944f71fd4e42a6ea46969c86e9f1) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.258160 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet dfd3e23d83a84648bd11cd258377aecb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.262184 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet ec13c5386abe414ba25589e03d5b850a with delete_type TABLET_DATA_DELETED (replaced by tablet 6699b50d87964f6a89c365171253ad25) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.262544 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 1eabfc95caa84afc9a8956a7ac13fea6 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b8e1d79d2a247dba25540daf6b81bf0) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.262666 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet f2d97084b5914350afbfda5c2a34bf9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.263175 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet a53f25688e3b45dca68984e2c543d094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.263715 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet 122c81787d5c43d28b1fb639e3542c76 with delete_type TABLET_DATA_DELETED (replaced by tablet b0b27d526b83492cbfa22604c943f76c) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.268568 2235 tablet_service.cc:1505] Processing CreateTablet for tablet a0b9626c8ed14636acdf0a286a709625 (DEFAULT_TABLE table=default.Table_73be634ceddf488ea518d690587f6cd0 [id=5e42165433a248a9bff99b74846d72e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.309870 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0b9626c8ed14636acdf0a286a709625. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.313198 3997 raft_consensus.cc:493] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.313344 3997 raft_consensus.cc:515] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.313894 3997 leader_election.cc:290] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.315021 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24f16e40a30c41788a6f9813c235c5ab" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:49.315570 2053 leader_election.cc:343] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.272982 4067 raft_consensus.cc:493] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.277702 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 9b3bb056c94149bbb9f3dbfeb6db0d5a (DEFAULT_TABLE table=default.Table_debeb38e035a435fa936ab479b12d30e [id=caf30a612cba4ce6954ab61eacc8ba00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.316457 4067 raft_consensus.cc:515] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.316645 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b3bb056c94149bbb9f3dbfeb6db0d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.317727 4067 leader_election.cc:290] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:49.278314 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 6699b50d87964f6a89c365171253ad25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.278710 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 209fa71af13142a58efb9f3dab3eecd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.321237 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "398c1f420974453b9b6b0fe17795378b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:49.321352 2377 ts_tablet_manager.cc:1950] T f437d8b5f5784e8fbec10d0700e90348 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:49.321436 2116 raft_consensus.cc:2468] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:49.279018 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e203fe0403594d30b44d531153d42a10" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.279318 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81775055d98349f696cdab6171ffd58b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.321980 2262 raft_consensus.cc:2468] T e203fe0403594d30b44d531153d42a10 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:49.296598 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85420a102dae4269811dbdc64aebb20a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.297030 4003 consensus_queue.cc:1048] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:49.322399 2053 leader_election.cc:343] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.322527 2053 leader_election.cc:304] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.322743 3997 raft_consensus.cc:2749] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.323033 2053 leader_election.cc:304] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.323467 3997 raft_consensus.cc:2804] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:49.323565 3997 raft_consensus.cc:493] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:49.323637 3997 raft_consensus.cc:3060] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:49.323879 2053 leader_election.cc:343] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.323969 2053 leader_election.cc:304] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.324277 4098 raft_consensus.cc:2749] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.322072 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 971) took 5696 ms. Trace:
I20251026 02:56:49.326320 2377 rpcz_store.cc:276] 1026 02:56:43.625632 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.625698 (+ 66us) service_pool.cc:225] Handling call
1026 02:56:49.060317 (+5434619us) ts_tablet_manager.cc:1151] Deleting tablet f437d8b5f5784e8fbec10d0700e90348
1026 02:56:49.060325 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.169448 (+109123us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.215358 (+ 45910us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.322056 (+106698us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5434361}
I20251026 02:56:49.326697 2377 tablet_replica.cc:333] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:49.326614 3997 raft_consensus.cc:515] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.326871 2377 raft_consensus.cc:2243] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:49.327133 3997 leader_election.cc:290] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.328845 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fc36553c686349288efdaf7986cd5302"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.329439 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc36553c686349288efdaf7986cd5302 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.329558 2195 leader_election.cc:304] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:49.330137 4003 raft_consensus.cc:2804] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:49.330209 4003 raft_consensus.cc:493] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:49.330276 4003 raft_consensus.cc:3060] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.330437 2376 rpcz_store.cc:276] 1026 02:56:46.344759 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.344817 (+ 58us) service_pool.cc:225] Handling call
1026 02:56:49.119300 (+2774483us) ts_tablet_manager.cc:1151] Deleting tablet ec13c5386abe414ba25589e03d5b850a
1026 02:56:49.119308 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.157745 (+ 38437us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.302662 (+144917us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.303800 (+ 1138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2774308}
I20251026 02:56:49.330999 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e203fe0403594d30b44d531153d42a10" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:49.331200 2260 raft_consensus.cc:3060] T e203fe0403594d30b44d531153d42a10 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.332939 4003 raft_consensus.cc:515] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.333024 2376 tablet_replica.cc:333] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:49.333221 2376 raft_consensus.cc:2243] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:49.333396 2376 raft_consensus.cc:2272] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:49.333674 4003 leader_election.cc:290] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:49.333704 2260 raft_consensus.cc:2468] T e203fe0403594d30b44d531153d42a10 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:49.334004 2376 ts_tablet_manager.cc:1916] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:49.334206 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "398c1f420974453b9b6b0fe17795378b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:49.334482 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "398c1f420974453b9b6b0fe17795378b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:49.334635 2116 raft_consensus.cc:3060] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:49.334738 2194 leader_election.cc:343] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.336802 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 983) took 5414 ms. Trace:
I20251026 02:56:49.336882 2372 rpcz_store.cc:276] 1026 02:56:43.922162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.922217 (+ 55us) service_pool.cc:225] Handling call
1026 02:56:49.160866 (+5238649us) ts_tablet_manager.cc:1151] Deleting tablet e89a98fcfdbb4011876b0ebc2af91f67
1026 02:56:49.160875 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.257441 (+ 96566us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.287507 (+ 30066us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.336779 (+ 49272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5238458}
I20251026 02:56:49.337184 2372 tablet_replica.cc:333] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:49.337355 2372 raft_consensus.cc:2243] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:49.337359 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet e89a98fcfdbb4011876b0ebc2af91f67 (table default.Table_53a2c0226802442a8954f3903443b574 [id=881224ae789e4daab936faafe4765729]) successfully deleted
I20251026 02:56:49.337491 2372 raft_consensus.cc:2272] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:49.337641 2116 raft_consensus.cc:2468] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:49.338003 2372 ts_tablet_manager.cc:1916] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:49.305120 2378 tablet_bootstrap.cc:492] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:49.307214 1915 leader_election.cc:304] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:49.338248 2378 ts_tablet_manager.cc:1403] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.122s user 0.002s sys 0.002s
I20251026 02:56:49.314587 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24f16e40a30c41788a6f9813c235c5ab" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.338495 2053 leader_election.cc:304] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:49.317456 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "398c1f420974453b9b6b0fe17795378b" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.327662 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e203fe0403594d30b44d531153d42a10" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:49.338956 2053 leader_election.cc:343] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.338953 2195 leader_election.cc:304] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.339061 2053 leader_election.cc:304] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.339160 4167 raft_consensus.cc:2749] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.339390 4098 raft_consensus.cc:2804] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:49.339881 4067 raft_consensus.cc:2804] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 02:56:49.340041 2194 leader_election.cc:343] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.340287 4067 raft_consensus.cc:697] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
W20251026 02:56:49.340881 2053 leader_election.cc:343] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.343488 4067 consensus_queue.cc:237] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.344115 3997 raft_consensus.cc:2749] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.352015 2375 tablet_bootstrap.cc:492] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:49.352258 2375 ts_tablet_manager.cc:1403] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.000s
I20251026 02:56:49.353029 2375 raft_consensus.cc:359] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.353304 2375 raft_consensus.cc:385] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.353403 2375 raft_consensus.cc:740] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.353674 2375 consensus_queue.cc:260] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.354318 2375 ts_tablet_manager.cc:1434] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:49.354416 3997 raft_consensus.cc:493] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.354547 3997 raft_consensus.cc:515] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.354734 1980 raft_consensus.cc:3060] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.355065 3997 leader_election.cc:290] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.356074 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892a5fab2af1432eb31d92d9859ea2a0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:49.356520 2053 leader_election.cc:343] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.358337 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892a5fab2af1432eb31d92d9859ea2a0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:49.358876 2053 leader_election.cc:343] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.359018 2053 leader_election.cc:304] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.359436 3997 raft_consensus.cc:2749] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:49.360304 1624 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu pid 1700: Timed out: CheckLeaks RPC to 127.1.150.61:37655 timed out after 1.000s (SENT)
I20251026 02:56:49.362313 2375 tablet_bootstrap.cc:492] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:49.364382 4098 raft_consensus.cc:697] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:49.364679 4098 consensus_queue.cc:237] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.368335 1624 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu with pid 1700
I20251026 02:56:49.384374 2378 raft_consensus.cc:359] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.385056 2378 raft_consensus.cc:385] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.385118 4167 raft_consensus.cc:493] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.385275 4167 raft_consensus.cc:515] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.385921 4167 leader_election.cc:290] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.386121 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet ec13c5386abe414ba25589e03d5b850a (table default.Table_3792bcbbc16b4be0b113cb5f07b9261b [id=8a3d1c20763445c499bd655eb99deb56]) successfully deleted
I20251026 02:56:49.386512 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a00bc04b3654f90af342986751344e6" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.386588 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a00bc04b3654f90af342986751344e6" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:49.386968 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet f437d8b5f5784e8fbec10d0700e90348 (table default.Table_b555f404492a47808d8a9a168a45ef2f [id=6185bafc43f340dfa6e187a6e5468874]) successfully deleted
I20251026 02:56:49.387600 1916 leader_election.cc:304] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:49.388258 4167 raft_consensus.cc:2749] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.399395 2377 raft_consensus.cc:2272] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:49.399992 2377 ts_tablet_manager.cc:1916] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:49.400996 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 200af441a3d24c3c8260acfa73cfe52c consensus state has no leader
W20251026 02:56:49.404280 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f105398b04454473b2a283e729eb2139 consensus state has no leader
I20251026 02:56:49.404625 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0aab08dd62de4321ba16eae6624f2d5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:49.405699 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_016a71d765444bc78bf9e3cf36571b0b" } modify_external_catalogs: true
I20251026 02:56:49.405866 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_016a71d765444bc78bf9e3cf36571b0b" } modify_external_catalogs: true
I20251026 02:56:49.406581 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 47bfff40c181432896d6898ea50cd025
I20251026 02:56:49.407205 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7807f6bf2766414497c3dce95b5319cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.407720 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7807f6bf2766414497c3dce95b5319cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.408784 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet f59ea356389346359d66b207c5cdb69f
I20251026 02:56:49.409366 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_342101e3599e42358d22806e5e119290" } new_table_name: "default.Table_33747351d5ee48bfb112182b05f47bee" modify_external_catalogs: true
I20251026 02:56:49.410885 4246 raft_consensus.cc:493] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.411017 4246 raft_consensus.cc:515] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.411579 4246 leader_election.cc:290] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.412966 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ea364c4a5cf4f3f99d9ff6904add697" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.413584 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet f59ea356389346359d66b207c5cdb69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:49.413937 2053 leader_election.cc:343] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.414186 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ea364c4a5cf4f3f99d9ff6904add697" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.414520 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 47bfff40c181432896d6898ea50cd025 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:49.415035 2053 leader_election.cc:343] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.414954 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_8d7bf2ec1d04457cba728889928fae0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:49.415133 2053 leader_election.cc:304] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.415494 4246 raft_consensus.cc:2749] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:49.415499 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d7bf2ec1d04457cba728889928fae0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:49.418313 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0aab08dd62de4321ba16eae6624f2d5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.418694 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 47bfff40c181432896d6898ea50cd025 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.419324 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 47bfff40c181432896d6898ea50cd025 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.419930 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet f59ea356389346359d66b207c5cdb69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.420282 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet f59ea356389346359d66b207c5cdb69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.420614 2381 tablet_bootstrap.cc:492] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:49.420787 2381 ts_tablet_manager.cc:1403] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.231s user 0.001s sys 0.004s
I20251026 02:56:49.421823 2381 raft_consensus.cc:359] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.422130 2381 raft_consensus.cc:385] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.422236 2381 raft_consensus.cc:740] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.422744 2381 consensus_queue.cc:260] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.423305 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet f7858b6a20814d5cb494be2a5001ae49
I20251026 02:56:49.385135 2378 raft_consensus.cc:740] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.423938 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7847c12059f440cf83e6f1c7dc0f6808
I20251026 02:56:49.425544 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 7847c12059f440cf83e6f1c7dc0f6808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.425592 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet f7858b6a20814d5cb494be2a5001ae49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.427147 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9615d980ff294c7f9cee5fd619547a01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.427747 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9615d980ff294c7f9cee5fd619547a01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.435377 2375 tablet_bootstrap.cc:654] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.435447 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d744c8c07d3c4f25b019bfe49bf82b68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.436033 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d744c8c07d3c4f25b019bfe49bf82b68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.438087 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet f7858b6a20814d5cb494be2a5001ae49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.440840 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 7847c12059f440cf83e6f1c7dc0f6808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.436318 2378 consensus_queue.cc:260] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.438462 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet f7858b6a20814d5cb494be2a5001ae49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.440201 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 7847c12059f440cf83e6f1c7dc0f6808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.444913 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e961fa54c9e84ce0bc37f0aa04d6e4a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.445602 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e961fa54c9e84ce0bc37f0aa04d6e4a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.447999 1981 raft_consensus.cc:3060] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.451910 4067 consensus_queue.cc:1048] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:49.456321 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_98834edef3e047759f8740f45604592c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:49.464905 2381 ts_tablet_manager.cc:1434] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.044s user 0.002s sys 0.000s
I20251026 02:56:49.465024 4246 raft_consensus.cc:493] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.465190 4246 raft_consensus.cc:515] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.465763 2378 ts_tablet_manager.cc:1434] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.127s user 0.002s sys 0.000s
I20251026 02:56:49.466105 4246 leader_election.cc:290] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.466523 4246 raft_consensus.cc:493] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.466625 4246 raft_consensus.cc:515] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.467141 4246 leader_election.cc:290] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.467489 4246 raft_consensus.cc:493] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.467582 4246 raft_consensus.cc:515] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.468015 4246 leader_election.cc:290] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:49.469184 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98834edef3e047759f8740f45604592c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.469036 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c966cae6886048a2b6ad3b8bd7007bcb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.469290 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8fe7825fa8d46669772cf84bc4b3afc" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.481982 2378 tablet_bootstrap.cc:492] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:49.482532 2053 leader_election.cc:343] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:49.482772 2053 leader_election.cc:343] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.472183 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aea01a248b345cf81d4d9862744eb78" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.484064 2378 tablet_bootstrap.cc:654] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.470299 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aea01a248b345cf81d4d9862744eb78" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:49.484472 2053 leader_election.cc:343] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.470546 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c966cae6886048a2b6ad3b8bd7007bcb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.484681 2381 tablet_bootstrap.cc:492] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:49.470566 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8fe7825fa8d46669772cf84bc4b3afc" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:49.484925 2053 leader_election.cc:343] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.485013 2053 leader_election.cc:304] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:49.485423 2053 leader_election.cc:343] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.485524 2053 leader_election.cc:304] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:49.485916 2053 leader_election.cc:343] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.486007 2375 tablet_bootstrap.cc:492] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:49.486078 3997 raft_consensus.cc:2749] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.486163 2375 ts_tablet_manager.cc:1403] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.124s user 0.003s sys 0.000s
I20251026 02:56:49.486250 3997 raft_consensus.cc:2749] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.486012 2053 leader_election.cc:304] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.486956 4246 raft_consensus.cc:2749] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.486817 2375 raft_consensus.cc:359] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.489599 2375 raft_consensus.cc:385] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.489677 2375 raft_consensus.cc:740] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.490543 2375 consensus_queue.cc:260] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.491056 2375 ts_tablet_manager.cc:1434] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251026 02:56:49.491358 2375 tablet_bootstrap.cc:492] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:49.492767 2375 tablet_bootstrap.cc:654] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.495337 4266 tablet.cc:1722] T 7abcdac4a70b48ad9fbba1bc8a60283b P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:49.501108 2375 tablet_bootstrap.cc:492] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:49.501438 2375 ts_tablet_manager.cc:1403] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20251026 02:56:49.502193 2375 raft_consensus.cc:359] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.502395 2375 raft_consensus.cc:385] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.502461 2375 raft_consensus.cc:740] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.506408 2375 consensus_queue.cc:260] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.513027 2381 tablet_bootstrap.cc:654] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.514789 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet f27da2b726344153b54c0187660ca39f
I20251026 02:56:49.517091 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet f27da2b726344153b54c0187660ca39f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.519059 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet f27da2b726344153b54c0187660ca39f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.519543 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet f27da2b726344153b54c0187660ca39f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.519557 4265 tablet.cc:1722] T 7abcdac4a70b48ad9fbba1bc8a60283b P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:49.523774 2376 ts_tablet_manager.cc:1929] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:49.530377 2375 ts_tablet_manager.cc:1434] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20251026 02:56:49.530706 2375 tablet_bootstrap.cc:492] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:49.535873 2195 consensus_peers.cc:597] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:49.543212 4098 raft_consensus.cc:493] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.543388 4098 raft_consensus.cc:515] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.544275 4098 leader_election.cc:290] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.544584 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24449cb9121642009dec41220e27b193" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:49.545001 2053 leader_election.cc:343] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.566231 2375 tablet_bootstrap.cc:654] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.567638 2381 tablet_bootstrap.cc:492] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:49.567863 2381 ts_tablet_manager.cc:1403] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.083s user 0.000s sys 0.003s
I20251026 02:56:49.568638 2381 raft_consensus.cc:359] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.568902 2381 raft_consensus.cc:385] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.569008 2381 raft_consensus.cc:740] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.569276 2381 consensus_queue.cc:260] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:49.572080 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06de594e17244f0d906ba296231358d4 has no consensus state
I20251026 02:56:49.572505 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2939f4b1afea4a4b8bbfe83ba4d58558" } new_table_name: "default.Table_65337bf4c4cb479fba072b3edcfd6e56" modify_external_catalogs: true
I20251026 02:56:49.573894 1655 master_service.cc:954] ReplaceTablet: received request to replace tablet c95cf788ecfd4ed491ff49f9f2dc8db3 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:49.574127 2376 log.cc:1199] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3e38f282e8414b7db2a6d5f4a8933e84
I20251026 02:56:49.574366 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 47bfff40c181432896d6898ea50cd025 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:49.574438 2376 ts_tablet_manager.cc:1950] T 3e38f282e8414b7db2a6d5f4a8933e84 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:49.575228 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1000) took 3230 ms. Trace:
I20251026 02:56:49.575315 2376 rpcz_store.cc:276] 1026 02:56:46.344867 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.345031 (+ 164us) service_pool.cc:225] Handling call
1026 02:56:49.332973 (+2987942us) ts_tablet_manager.cc:1151] Deleting tablet 3e38f282e8414b7db2a6d5f4a8933e84
1026 02:56:49.332983 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.477174 (+144191us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.523668 (+ 46494us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.575194 (+ 51526us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2987820}
I20251026 02:56:49.575687 2376 tablet_replica.cc:333] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:49.575871 2376 raft_consensus.cc:2243] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:49.576000 2376 raft_consensus.cc:2272] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:49.575965 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3e38f282e8414b7db2a6d5f4a8933e84 (table default.Table_3778c4a74e2f42e49c437a2aa3c5f22b [id=e2f6dfa3686b41ddbc9f3b0205c7f05f]) successfully deleted
I20251026 02:56:49.576577 2376 ts_tablet_manager.cc:1916] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:49.545746 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24449cb9121642009dec41220e27b193" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.560452 4271 tablet.cc:1722] T 7abcdac4a70b48ad9fbba1bc8a60283b P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251026 02:56:49.578779 2053 leader_election.cc:343] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.578928 2053 leader_election.cc:304] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.574013 2378 tablet_bootstrap.cc:492] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:49.579394 4098 raft_consensus.cc:2749] T 24449cb9121642009dec41220e27b193 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.579414 2378 ts_tablet_manager.cc:1403] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.000s
I20251026 02:56:49.580245 2378 raft_consensus.cc:359] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.580488 2378 raft_consensus.cc:385] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.580610 2378 raft_consensus.cc:740] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.580888 2378 consensus_queue.cc:260] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.581574 2378 ts_tablet_manager.cc:1434] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 02:56:49.581868 2378 tablet_bootstrap.cc:492] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:49.583557 2378 tablet_bootstrap.cc:654] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.586670 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_be8c60cde79445d79d373c6fa4c0f745" } new_table_name: "default.Table_5800c367f03d4a7fb9475a364d30e14b" modify_external_catalogs: true
I20251026 02:56:49.588120 2375 tablet_bootstrap.cc:492] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:49.588282 2375 ts_tablet_manager.cc:1403] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20251026 02:56:49.588930 2375 raft_consensus.cc:359] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.589092 2375 raft_consensus.cc:385] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.589141 2375 raft_consensus.cc:740] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.589329 2375 consensus_queue.cc:260] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:49.593971 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61d46c20e3254369a69dab02b52e12ff consensus state has no leader
I20251026 02:56:49.594522 2375 ts_tablet_manager.cc:1434] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251026 02:56:49.594595 2381 ts_tablet_manager.cc:1434] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20251026 02:56:49.594817 2375 tablet_bootstrap.cc:492] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:49.596242 2375 tablet_bootstrap.cc:654] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.599956 4265 tablet.cc:1722] T 386e1bd4635b4a7e8e7f249b577fd978 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:49.578774 2372 ts_tablet_manager.cc:1929] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:49.600818 2372 log.cc:1199] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/dfd3e23d83a84648bd11cd258377aecb
I20251026 02:56:49.599014 4266 tablet.cc:1722] T 386e1bd4635b4a7e8e7f249b577fd978 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:49.601848 2381 tablet_bootstrap.cc:492] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:49.598758 2377 ts_tablet_manager.cc:1929] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:49.602075 2377 log.cc:1199] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/aa10cd990a944eab8d9eb66aaf20080c
I20251026 02:56:49.602547 2377 ts_tablet_manager.cc:1950] T aa10cd990a944eab8d9eb66aaf20080c P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:49.604981 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 972) took 5979 ms. Trace:
I20251026 02:56:49.601251 2372 ts_tablet_manager.cc:1950] T dfd3e23d83a84648bd11cd258377aecb P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:49.614565 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 984) took 5656 ms. Trace:
I20251026 02:56:49.614663 2372 rpcz_store.cc:276] 1026 02:56:43.958070 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.958127 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:49.337139 (+5379012us) ts_tablet_manager.cc:1151] Deleting tablet dfd3e23d83a84648bd11cd258377aecb
1026 02:56:49.337149 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.426875 (+ 89726us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.578666 (+151791us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.614542 (+ 35876us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5378770}
I20251026 02:56:49.615028 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 986) took 5461 ms. Trace:
I20251026 02:56:49.615127 2372 rpcz_store.cc:276] 1026 02:56:44.153884 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.153954 (+ 70us) service_pool.cc:225] Handling call
1026 02:56:49.614915 (+5460961us) ts_tablet_manager.cc:1151] Deleting tablet 0afb6f04f5ac463780fe8f522bf2fe23
1026 02:56:49.614924 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.615015 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5432155}
I20251026 02:56:49.615466 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 987) took 5449 ms. Trace:
I20251026 02:56:49.617147 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet dfd3e23d83a84648bd11cd258377aecb (table default.Table_f4bddfca37d64369b99a1fdc0102ca35 [id=434c941e53194bcf9a53a590775d6a6d]) successfully deleted
W20251026 02:56:49.617480 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0afb6f04f5ac463780fe8f522bf2fe23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0afb6f04f5ac463780fe8f522bf2fe23 already in progress: creating tablet
I20251026 02:56:49.618831 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d2755d308fc34447b8e72260cb3c0f76" } new_table_name: "default.Table_dded0e04841c47239eb9ffe89f7ea93b" modify_external_catalogs: true
I20251026 02:56:49.619383 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet 5b8e1d79d2a247dba25540daf6b81bf0 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:49.620298 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1a8082b54aaf4bb4809829906a72db16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.620797 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a8082b54aaf4bb4809829906a72db16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.621162 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_43ea6a50a51e416da1649a35e72a7283" } new_table_name: "default.Table_3e40ba9b6c944a3899576ca411fc1ffb" modify_external_catalogs: true
I20251026 02:56:49.622015 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_dbd6c27f936b44f48242fff9d1b6ec82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.622509 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbd6c27f936b44f48242fff9d1b6ec82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.623546 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_49ed105063f845bd92ee37a1ce8d97a7" } modify_external_catalogs: true
I20251026 02:56:49.623705 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_49ed105063f845bd92ee37a1ce8d97a7" } modify_external_catalogs: true
W20251026 02:56:49.624841 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11f4c327caa644d89098496bc324b976 consensus state has no leader
I20251026 02:56:49.625247 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_33747351d5ee48bfb112182b05f47bee" } modify_external_catalogs: true
I20251026 02:56:49.625393 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_33747351d5ee48bfb112182b05f47bee" } modify_external_catalogs: true
I20251026 02:56:49.628305 2377 rpcz_store.cc:276] 1026 02:56:43.625760 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.626002 (+ 242us) service_pool.cc:225] Handling call
1026 02:56:49.326649 (+5700647us) ts_tablet_manager.cc:1151] Deleting tablet aa10cd990a944eab8d9eb66aaf20080c
1026 02:56:49.326659 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.499606 (+172947us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.544196 (+ 44590us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.604962 (+ 60766us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5700495}
I20251026 02:56:49.623148 4069 raft_consensus.cc:493] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.629537 4069 raft_consensus.cc:515] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.630101 4069 leader_election.cc:290] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:49.627643 2194 consensus_peers.cc:597] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:49.613358 4258 raft_consensus.cc:493] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.630812 4258 raft_consensus.cc:515] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.631378 4258 leader_election.cc:290] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:49.640487 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet aa10cd990a944eab8d9eb66aaf20080c (table default.Table_42c4c5e055054536b2b541711288a481 [id=503128cdfe654cfb9713c8e241eac740]) successfully deleted
I20251026 02:56:49.640585 2377 tablet_replica.cc:333] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:49.640806 2377 raft_consensus.cc:2243] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:49.641170 2377 raft_consensus.cc:2272] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:49.641803 2377 ts_tablet_manager.cc:1916] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:49.632182 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "209fa71af13142a58efb9f3dab3eecd3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:49.615538 2372 rpcz_store.cc:276] 1026 02:56:44.166171 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.166369 (+ 198us) service_pool.cc:225] Handling call
1026 02:56:49.615356 (+5448987us) ts_tablet_manager.cc:1151] Deleting tablet 146d6f13b9ef47f29711a102791bab5a
1026 02:56:49.615365 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.615453 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5432444}
I20251026 02:56:49.644841 2109 raft_consensus.cc:2468] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:49.645064 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 988) took 5478 ms. Trace:
I20251026 02:56:49.645139 2372 rpcz_store.cc:276] 1026 02:56:44.166314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.166571 (+ 257us) service_pool.cc:225] Handling call
1026 02:56:49.644933 (+5478362us) ts_tablet_manager.cc:1151] Deleting tablet 0c60d8227eed417595ef68d44c68ebcf
1026 02:56:49.644945 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.645050 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5461969}
I20251026 02:56:49.645285 2195 leader_election.cc:304] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
W20251026 02:56:49.645371 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 146d6f13b9ef47f29711a102791bab5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 146d6f13b9ef47f29711a102791bab5a already in progress: creating tablet
I20251026 02:56:49.645452 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 989) took 5467 ms. Trace:
I20251026 02:56:49.631772 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6699b50d87964f6a89c365171253ad25" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.632782 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6699b50d87964f6a89c365171253ad25" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:49.645702 2116 raft_consensus.cc:2468] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:49.632851 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "209fa71af13142a58efb9f3dab3eecd3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:49.646013 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0c60d8227eed417595ef68d44c68ebcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c60d8227eed417595ef68d44c68ebcf already in progress: creating tablet
I20251026 02:56:49.646036 2195 leader_election.cc:304] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:49.646273 4255 raft_consensus.cc:2804] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:49.646304 4069 raft_consensus.cc:2804] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:49.646349 4255 raft_consensus.cc:493] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:49.646389 4069 raft_consensus.cc:493] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:49.646425 4255 raft_consensus.cc:3060] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.646445 4069 raft_consensus.cc:3060] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.644452 4271 tablet.cc:1722] T 386e1bd4635b4a7e8e7f249b577fd978 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:49.648916 4069 raft_consensus.cc:515] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.649029 4255 raft_consensus.cc:515] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.649708 4255 leader_election.cc:290] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:49.649761 4069 leader_election.cc:290] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:49.650599 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6699b50d87964f6a89c365171253ad25" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:49.650775 2110 raft_consensus.cc:3060] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.653505 2110 raft_consensus.cc:2468] T 6699b50d87964f6a89c365171253ad25 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:49.653924 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "209fa71af13142a58efb9f3dab3eecd3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:49.654093 2109 raft_consensus.cc:3060] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.657070 2195 leader_election.cc:304] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:49.657424 4069 raft_consensus.cc:2804] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:49.657538 4069 raft_consensus.cc:697] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:49.657752 4069 consensus_queue.cc:237] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.658932 4098 raft_consensus.cc:493] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.659055 4098 raft_consensus.cc:515] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.659632 4098 leader_election.cc:290] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.660169 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1b6613742a24c21a1db6ebe858a846f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.660348 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1b6613742a24c21a1db6ebe858a846f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:49.660565 2194 leader_election.cc:343] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:49.660885 2194 leader_election.cc:343] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.660907 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "209fa71af13142a58efb9f3dab3eecd3" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:49.661147 2053 leader_election.cc:343] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:49.661371 2053 leader_election.cc:343] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.661458 2053 leader_election.cc:304] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.661789 2109 raft_consensus.cc:2468] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:49.662045 4098 raft_consensus.cc:2749] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:49.662274 2194 leader_election.cc:343] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.662395 2195 leader_election.cc:304] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.662721 4258 raft_consensus.cc:2804] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:49.662842 4258 raft_consensus.cc:697] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:49.663075 4258 consensus_queue.cc:237] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.664811 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6699b50d87964f6a89c365171253ad25" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:49.665480 2194 leader_election.cc:343] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:49.667682 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3183f909b0334d95b9ff9417c2954766 consensus state has no leader
I20251026 02:56:49.668118 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_efc39d27d9d5438ea04cef669a110c8a" } modify_external_catalogs: true
I20251026 02:56:49.668325 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_efc39d27d9d5438ea04cef669a110c8a" } modify_external_catalogs: true
I20251026 02:56:49.645506 2372 rpcz_store.cc:276] 1026 02:56:44.178293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.178349 (+ 56us) service_pool.cc:225] Handling call
1026 02:56:49.645356 (+5467007us) ts_tablet_manager.cc:1151] Deleting tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1
1026 02:56:49.645367 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.645439 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5461990}
I20251026 02:56:49.668787 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 990) took 5490 ms. Trace:
W20251026 02:56:49.668896 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 already in progress: creating tablet
I20251026 02:56:49.668871 2372 rpcz_store.cc:276] 1026 02:56:44.178403 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.178548 (+ 145us) service_pool.cc:225] Handling call
1026 02:56:49.668657 (+5490109us) ts_tablet_manager.cc:1151] Deleting tablet 63f1abc4477f4198ad2cc8481d69df73
1026 02:56:49.668669 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.668771 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5485237}
I20251026 02:56:49.669186 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 991) took 5490 ms. Trace:
W20251026 02:56:49.669279 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 63f1abc4477f4198ad2cc8481d69df73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63f1abc4477f4198ad2cc8481d69df73 already in progress: creating tablet
I20251026 02:56:49.669260 2372 rpcz_store.cc:276] 1026 02:56:44.178543 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.178701 (+ 158us) service_pool.cc:225] Handling call
1026 02:56:49.669092 (+5490391us) ts_tablet_manager.cc:1151] Deleting tablet 5eda85f2d53a43d68907c467c7da9bd6
1026 02:56:49.669104 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.669173 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5485636}
I20251026 02:56:49.669382 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_46eca4a0a83f4d3091c55183b402758d" } modify_external_catalogs: true
I20251026 02:56:49.669535 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 992) took 5453 ms. Trace:
I20251026 02:56:49.669541 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_46eca4a0a83f4d3091c55183b402758d" } modify_external_catalogs: true
I20251026 02:56:49.669597 2372 rpcz_store.cc:276] 1026 02:56:44.216256 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.216313 (+ 57us) service_pool.cc:225] Handling call
1026 02:56:49.669442 (+5453129us) ts_tablet_manager.cc:1151] Deleting tablet 146d6f13b9ef47f29711a102791bab5a
1026 02:56:49.669451 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.669519 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5452910}
I20251026 02:56:49.669864 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1005) took 4938 ms. Trace:
I20251026 02:56:49.669925 2372 rpcz_store.cc:276] 1026 02:56:44.731803 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.731892 (+ 89us) service_pool.cc:225] Handling call
1026 02:56:49.669776 (+4937884us) ts_tablet_manager.cc:1151] Deleting tablet e45600a92f8e46b096badd700779856d
1026 02:56:49.669784 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.669849 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4937667}
W20251026 02:56:49.670042 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 5eda85f2d53a43d68907c467c7da9bd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5eda85f2d53a43d68907c467c7da9bd6 already in progress: creating tablet
I20251026 02:56:49.670197 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1035) took 4428 ms. Trace:
W20251026 02:56:49.670258 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 146d6f13b9ef47f29711a102791bab5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 146d6f13b9ef47f29711a102791bab5a already in progress: creating tablet
I20251026 02:56:49.670284 2372 rpcz_store.cc:276] 1026 02:56:45.241555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.318843 (+ 77288us) service_pool.cc:225] Handling call
1026 02:56:49.670115 (+4351272us) ts_tablet_manager.cc:1151] Deleting tablet 3865399cf6934bc18e16ae2cb4887197
1026 02:56:49.670123 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.670184 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4351115}
I20251026 02:56:49.670531 2372 tablet_replica.cc:333] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:49.670722 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet e45600a92f8e46b096badd700779856d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e45600a92f8e46b096badd700779856d already in progress: creating tablet
I20251026 02:56:49.670807 2372 raft_consensus.cc:2243] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251026 02:56:49.670934 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 3865399cf6934bc18e16ae2cb4887197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3865399cf6934bc18e16ae2cb4887197 already in progress: creating tablet
I20251026 02:56:49.670962 2372 raft_consensus.cc:2272] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:49.675313 1647 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 11f4c327caa644d89098496bc324b976 (table default.Table_5800c367f03d4a7fb9475a364d30e14b [id=fff9b966161444b990693ad4adbfcbc8], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 11f4c327caa644d89098496bc324b976 consensus state has no leader
I20251026 02:56:49.603422 2381 tablet_bootstrap.cc:654] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:49.680203 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cef68c4d9a3e4a85acdab446bd6082ce consensus state has no leader
I20251026 02:56:49.680682 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_cedb95f3248844edaeb1737e91c70c9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.681284 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cedb95f3248844edaeb1737e91c70c9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.682050 2376 ts_tablet_manager.cc:1929] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:49.682173 2376 log.cc:1199] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/c31d2704d1f14ddead69c794cc631ec1
I20251026 02:56:49.682591 2376 ts_tablet_manager.cc:1950] T c31d2704d1f14ddead69c794cc631ec1 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:49.682937 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1002) took 3282 ms. Trace:
I20251026 02:56:49.683019 2376 rpcz_store.cc:276] 1026 02:56:46.400448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.412626 (+ 12178us) service_pool.cc:225] Handling call
1026 02:56:49.575639 (+3163013us) ts_tablet_manager.cc:1151] Deleting tablet c31d2704d1f14ddead69c794cc631ec1
1026 02:56:49.575649 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.630049 (+ 54400us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.681921 (+ 51872us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.682918 (+ 997us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3162754}
I20251026 02:56:49.683537 2376 tablet_replica.cc:333] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:49.684252 2372 ts_tablet_manager.cc:1916] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:49.687587 2378 tablet_bootstrap.cc:492] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:49.687762 2378 ts_tablet_manager.cc:1403] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.106s user 0.001s sys 0.002s
I20251026 02:56:49.688666 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet c31d2704d1f14ddead69c794cc631ec1 (table default.Table_b6138cb4ba3c4b70bcec42f1ce989668 [id=9b9d43027de84287b1e2fd35f5e4d5f5]) successfully deleted
I20251026 02:56:49.689038 2378 raft_consensus.cc:359] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.689265 2378 raft_consensus.cc:385] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.689329 2378 raft_consensus.cc:740] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.689600 2378 consensus_queue.cc:260] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.691231 1657 master_service.cc:954] ReplaceTablet: received request to replace tablet 85420a102dae4269811dbdc64aebb20a from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:49.691671 4167 raft_consensus.cc:493] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.691825 4167 raft_consensus.cc:515] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.692655 4167 leader_election.cc:290] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.691826 2378 ts_tablet_manager.cc:1434] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251026 02:56:49.693219 2378 tablet_bootstrap.cc:492] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:49.693243 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dbbfb77851d443f8cf4d798f3e07e59" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:49.693427 2109 raft_consensus.cc:2468] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:49.693789 1916 leader_election.cc:304] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:49.694064 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dbbfb77851d443f8cf4d798f3e07e59" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.694499 4167 raft_consensus.cc:2804] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:49.694571 4167 raft_consensus.cc:493] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:49.694628 4167 raft_consensus.cc:3060] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:49.695245 1915 leader_election.cc:343] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.696257 4069 consensus_queue.cc:1048] T 41a1442fffe54add93666e3c78a0cc58 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:49.698670 4167 raft_consensus.cc:515] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.700031 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dbbfb77851d443f8cf4d798f3e07e59" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:49.699960 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dbbfb77851d443f8cf4d798f3e07e59" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:49.700380 2109 raft_consensus.cc:3060] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:49.700560 1915 leader_election.cc:343] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.700944 4167 leader_election.cc:290] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.701249 4167 raft_consensus.cc:493] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.701505 4167 raft_consensus.cc:515] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.702847 4167 leader_election.cc:290] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:49.703480 1916 leader_election.cc:343] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 7f22c6c163c34ed48a0b7eeb5a574e76
W20251026 02:56:49.703823 1915 leader_election.cc:343] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 7f22c6c163c34ed48a0b7eeb5a574e76
I20251026 02:56:49.703924 1915 leader_election.cc:304] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:49.703955 2109 raft_consensus.cc:2468] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:49.704385 4167 raft_consensus.cc:2749] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.704466 1916 leader_election.cc:304] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8; no voters: 696476265495492790739ec126a08539
I20251026 02:56:49.704756 4167 raft_consensus.cc:2804] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:49.704965 4167 raft_consensus.cc:697] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:49.705452 4167 consensus_queue.cc:237] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.710325 2378 tablet_bootstrap.cc:654] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.717238 4098 raft_consensus.cc:493] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.717407 4098 raft_consensus.cc:515] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.717796 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4e171e396e77413bb4d15fb2bcf9b0e5" } modify_external_catalogs: true
I20251026 02:56:49.717988 4098 leader_election.cc:290] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.717986 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4e171e396e77413bb4d15fb2bcf9b0e5" } modify_external_catalogs: true
I20251026 02:56:49.718423 4098 raft_consensus.cc:493] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.718537 4098 raft_consensus.cc:515] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.718986 4098 leader_election.cc:290] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.719430 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "351cd60b076a49949f03640c981845c4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.719761 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db94738da3554fbda64c23fde3ef6c1b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:49.719923 2053 leader_election.cc:343] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:49.720317 2053 leader_election.cc:343] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.720373 4258 consensus_queue.cc:1048] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:49.721567 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db94738da3554fbda64c23fde3ef6c1b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:49.722041 2053 leader_election.cc:343] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.722136 2053 leader_election.cc:304] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.722321 4098 raft_consensus.cc:2749] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.722311 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "351cd60b076a49949f03640c981845c4" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:49.722831 2053 leader_election.cc:343] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.722925 2053 leader_election.cc:304] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.723129 4098 raft_consensus.cc:2749] T 351cd60b076a49949f03640c981845c4 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.741276 2376 raft_consensus.cc:2243] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:49.746357 1975 raft_consensus.cc:3060] T c0bacd15b41149b5856d20068711b432 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.748325 2381 tablet_bootstrap.cc:492] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:49.748571 2381 ts_tablet_manager.cc:1403] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.147s user 0.001s sys 0.003s
I20251026 02:56:49.749622 2381 raft_consensus.cc:359] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.749864 2381 raft_consensus.cc:385] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.750090 2381 raft_consensus.cc:740] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.750329 2381 consensus_queue.cc:260] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.750795 2381 ts_tablet_manager.cc:1434] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:49.751021 2378 tablet_bootstrap.cc:492] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:49.751150 2381 tablet_bootstrap.cc:492] T e89a98fcfdbb4011876b0ebc2af91f67 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:49.751164 2378 ts_tablet_manager.cc:1403] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.001s
I20251026 02:56:49.751902 2378 raft_consensus.cc:359] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.752059 2378 raft_consensus.cc:385] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.752113 2378 raft_consensus.cc:740] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.752924 2381 tablet_bootstrap.cc:654] T e89a98fcfdbb4011876b0ebc2af91f67 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.756816 2378 consensus_queue.cc:260] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.757340 2378 ts_tablet_manager.cc:1434] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20251026 02:56:49.757588 2378 tablet_bootstrap.cc:492] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:49.758827 2377 ts_tablet_manager.cc:1929] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:49.759138 2378 tablet_bootstrap.cc:654] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.760445 2376 raft_consensus.cc:2272] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:49.761101 2377 log.cc:1199] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/86e483c52e514f1da54335479f59d68e
I20251026 02:56:49.761132 2376 ts_tablet_manager.cc:1916] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:49.761628 2377 ts_tablet_manager.cc:1950] T 86e483c52e514f1da54335479f59d68e P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:49.763698 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 974) took 6100 ms. Trace:
I20251026 02:56:49.769232 2377 rpcz_store.cc:276] 1026 02:56:43.662769 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.663216 (+ 447us) service_pool.cc:225] Handling call
1026 02:56:49.640525 (+5977309us) ts_tablet_manager.cc:1151] Deleting tablet 86e483c52e514f1da54335479f59d68e
1026 02:56:49.640537 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.730115 (+ 89578us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.758722 (+ 28607us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.763677 (+ 4955us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5977099}
I20251026 02:56:49.773458 2378 tablet_bootstrap.cc:492] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:49.773607 2378 ts_tablet_manager.cc:1403] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20251026 02:56:49.774339 2378 raft_consensus.cc:359] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.774504 2378 raft_consensus.cc:385] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.777065 2375 tablet_bootstrap.cc:492] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:49.777240 2375 ts_tablet_manager.cc:1403] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.182s user 0.008s sys 0.007s
I20251026 02:56:49.777909 2375 raft_consensus.cc:359] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.778062 2375 raft_consensus.cc:385] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.778151 2375 raft_consensus.cc:740] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.778332 2375 consensus_queue.cc:260] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.774554 2378 raft_consensus.cc:740] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.779662 2378 consensus_queue.cc:260] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.779876 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 06de594e17244f0d906ba296231358d4
I20251026 02:56:49.780323 2378 ts_tablet_manager.cc:1434] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251026 02:56:49.780599 2378 tablet_bootstrap.cc:492] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:49.780615 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9ad5fd6d3de34e97b6c8f52360294bb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.781148 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ad5fd6d3de34e97b6c8f52360294bb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.782584 2378 tablet_bootstrap.cc:654] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.784567 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 06de594e17244f0d906ba296231358d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.786573 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 06de594e17244f0d906ba296231358d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.791982 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_53d4276e57e646faa0ed4bfab8ae8e82" } new_table_name: "default.Table_cb72a739fdba4ee994d499c507e4816a" modify_external_catalogs: true
W20251026 02:56:49.799208 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95a4aa0abc0b455186482b0a85178db9 consensus state has no leader
I20251026 02:56:49.799664 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b8b1c6cdc277442094096a590903a0ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.800562 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8b1c6cdc277442094096a590903a0ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.801524 2372 ts_tablet_manager.cc:1929] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:49.801640 2372 log.cc:1199] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/6ce4092b7ccc4888b5cbe3b38bee2e36
I20251026 02:56:49.801915 2372 ts_tablet_manager.cc:1950] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:49.779994 2194 consensus_peers.cc:597] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:49.786041 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 06de594e17244f0d906ba296231358d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.806846 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_738af4cbe3c5484ebb053c840d593c22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.807893 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_738af4cbe3c5484ebb053c840d593c22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.810629 4069 consensus_queue.cc:1048] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:49.811013 2375 ts_tablet_manager.cc:1434] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20251026 02:56:49.811317 2375 tablet_bootstrap.cc:492] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:49.812980 2375 tablet_bootstrap.cc:654] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.804138 2381 tablet_bootstrap.cc:492] T e89a98fcfdbb4011876b0ebc2af91f67 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:49.815730 4098 raft_consensus.cc:493] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.815852 4098 raft_consensus.cc:515] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.815649 2381 ts_tablet_manager.cc:1403] T e89a98fcfdbb4011876b0ebc2af91f67 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.000s
I20251026 02:56:49.816396 4098 leader_election.cc:290] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.816916 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfac07917ee34e6dba3fac5e081f6b43" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.816972 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfac07917ee34e6dba3fac5e081f6b43" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.817094 1975 raft_consensus.cc:2468] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:49.817416 2053 leader_election.cc:304] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8; no voters:
W20251026 02:56:49.817665 2053 leader_election.cc:343] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.817896 4098 raft_consensus.cc:2804] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:49.817947 4098 raft_consensus.cc:493] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:49.817992 4098 raft_consensus.cc:3060] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.818207 2381 raft_consensus.cc:359] T e89a98fcfdbb4011876b0ebc2af91f67 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.818343 2381 raft_consensus.cc:385] T e89a98fcfdbb4011876b0ebc2af91f67 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.818392 2381 raft_consensus.cc:740] T e89a98fcfdbb4011876b0ebc2af91f67 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:49.818100 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_773097afa38549ee9cf02a6349454c59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:49.818579 2381 consensus_queue.cc:260] T e89a98fcfdbb4011876b0ebc2af91f67 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:49.818868 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_773097afa38549ee9cf02a6349454c59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.821357 4098 raft_consensus.cc:515] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.821630 4167 consensus_queue.cc:1048] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:49.822950 4098 leader_election.cc:290] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.822937 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfac07917ee34e6dba3fac5e081f6b43" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
W20251026 02:56:49.823411 2053 leader_election.cc:343] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.823734 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfac07917ee34e6dba3fac5e081f6b43" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:49.823968 1975 raft_consensus.cc:3060] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.825862 2381 ts_tablet_manager.cc:1434] T e89a98fcfdbb4011876b0ebc2af91f67 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251026 02:56:49.826607 2381 tablet_bootstrap.cc:492] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:49.827368 1975 raft_consensus.cc:2468] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:49.827914 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 86e483c52e514f1da54335479f59d68e (table default.Table_6d91747dd6164f0d84509497a93aeeea [id=f33faf56d22346bdaa270b22e673315a]) successfully deleted
I20251026 02:56:49.828815 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 975) took 6153 ms. Trace:
I20251026 02:56:49.828474 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_67618fda8a6d4b7a85112555b0961143"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.829422 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67618fda8a6d4b7a85112555b0961143 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.832362 2377 rpcz_store.cc:276] 1026 02:56:43.675571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.675813 (+ 242us) service_pool.cc:225] Handling call
1026 02:56:49.828693 (+6152880us) ts_tablet_manager.cc:1151] Deleting tablet 8fdadd053cc34aeca0d491eb3200207f
1026 02:56:49.828702 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.828799 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6152667}
I20251026 02:56:49.832592 1662 catalog_manager.cc:5649] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:49.833006 1662 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 398c1f420974453b9b6b0fe17795378b (table default.Table_a92e68d05f4b455d810180ee4cc5874d [id=7f64fbec3582415a8924d01e14856a56]). Expected version 1 got 0
W20251026 02:56:49.833220 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 8fdadd053cc34aeca0d491eb3200207f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fdadd053cc34aeca0d491eb3200207f already in progress: creating tablet
I20251026 02:56:49.833220 4098 raft_consensus.cc:493] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.833366 4098 raft_consensus.cc:515] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.834873 4098 leader_election.cc:290] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.834932 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09f5265d85c6496d85c555b2b847373a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.834295 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09f5265d85c6496d85c555b2b847373a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:49.835628 2053 leader_election.cc:343] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:49.835834 2053 leader_election.cc:343] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.835916 2053 leader_election.cc:304] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.836560 4246 raft_consensus.cc:2749] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.836565 2053 leader_election.cc:304] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8; no voters: 696476265495492790739ec126a08539
I20251026 02:56:49.836712 4246 raft_consensus.cc:493] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.836793 4246 raft_consensus.cc:515] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.836884 3997 raft_consensus.cc:2804] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:49.836957 3997 raft_consensus.cc:697] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:49.837147 3997 consensus_queue.cc:237] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.837240 4246 leader_election.cc:290] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.838729 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f0124f125c40d291c2211fd95c71c5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.839313 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f0124f125c40d291c2211fd95c71c5" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:49.839457 2053 leader_election.cc:343] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:49.839879 2053 leader_election.cc:343] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.839990 2053 leader_election.cc:304] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.840265 4246 raft_consensus.cc:2749] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.841138 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 2386155307964833b90565b6da44ab8b
I20251026 02:56:49.842020 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_eb612503151340bf85e80ad83525e96d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.842602 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb612503151340bf85e80ad83525e96d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.843938 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 2386155307964833b90565b6da44ab8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.849316 4168 raft_consensus.cc:493] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.849557 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 2386155307964833b90565b6da44ab8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.849516 4168 raft_consensus.cc:515] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.850215 4168 leader_election.cc:290] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:49.851585 1916 leader_election.cc:343] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: f2d97084b5914350afbfda5c2a34bf9a
I20251026 02:56:49.855404 2381 tablet_bootstrap.cc:654] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:49.855894 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d97084b5914350afbfda5c2a34bf9a" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.855948 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 2386155307964833b90565b6da44ab8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.856096 2262 raft_consensus.cc:2468] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:49.856496 1915 leader_election.cc:304] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:49.857690 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1036) took 4573 ms. Trace:
I20251026 02:56:49.857806 2372 rpcz_store.cc:276] 1026 02:56:45.284318 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.319007 (+ 34689us) service_pool.cc:225] Handling call
1026 02:56:49.670491 (+4351484us) ts_tablet_manager.cc:1151] Deleting tablet 6ce4092b7ccc4888b5cbe3b38bee2e36
1026 02:56:49.670499 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.764807 (+ 94308us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.801410 (+ 36603us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.857669 (+ 56259us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46439,"tablet-delete.queue_time_us":4351397}
I20251026 02:56:49.858291 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1038) took 4356 ms. Trace:
I20251026 02:56:49.858351 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 (table default.Table_a27cf0a8326c4683a305ae448369f2cd [id=d1ff1ff682064a5e96b158e9c64bdeb0]) successfully deleted
I20251026 02:56:49.858381 2372 rpcz_store.cc:276] 1026 02:56:45.501726 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.502491 (+ 765us) service_pool.cc:225] Handling call
1026 02:56:49.858132 (+4355641us) ts_tablet_manager.cc:1151] Deleting tablet 94d5faa4584a4e36b38bdd552d641994
1026 02:56:49.858141 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.858230 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4355404}
I20251026 02:56:49.858700 2372 tablet_replica.cc:333] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
W20251026 02:56:49.858811 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 94d5faa4584a4e36b38bdd552d641994 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94d5faa4584a4e36b38bdd552d641994 already in progress: creating tablet
I20251026 02:56:49.859900 2377 tablet_replica.cc:333] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:49.860075 2377 raft_consensus.cc:2243] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:49.860252 2377 raft_consensus.cc:2272] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:49.860828 2377 ts_tablet_manager.cc:1916] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:49.860664 1656 catalog_manager.cc:5649] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:49.861323 4167 raft_consensus.cc:2804] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:49.861402 4167 raft_consensus.cc:493] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:49.861459 4167 raft_consensus.cc:3060] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.862524 2372 raft_consensus.cc:2243] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:49.862705 2372 raft_consensus.cc:2272] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:49.863227 2372 ts_tablet_manager.cc:1916] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:49.864079 4167 raft_consensus.cc:515] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.865506 4167 leader_election.cc:290] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:49.866910 1916 leader_election.cc:343] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: f2d97084b5914350afbfda5c2a34bf9a
I20251026 02:56:49.866868 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d97084b5914350afbfda5c2a34bf9a" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:49.867064 2262 raft_consensus.cc:3060] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.868103 2376 ts_tablet_manager.cc:1929] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:49.868185 2376 log.cc:1199] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/86e483c52e514f1da54335479f59d68e
I20251026 02:56:49.868525 2376 ts_tablet_manager.cc:1950] T 86e483c52e514f1da54335479f59d68e P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:49.868788 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1004) took 3404 ms. Trace:
I20251026 02:56:49.868851 2376 rpcz_store.cc:276] 1026 02:56:46.463934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.464011 (+ 77us) service_pool.cc:225] Handling call
1026 02:56:49.683475 (+3219464us) ts_tablet_manager.cc:1151] Deleting tablet 86e483c52e514f1da54335479f59d68e
1026 02:56:49.683486 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.814060 (+130574us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.868018 (+ 53958us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.868774 (+ 756us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57331,"tablet-delete.queue_time_us":3219111}
I20251026 02:56:49.869632 2262 raft_consensus.cc:2468] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:49.869992 1915 leader_election.cc:304] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:49.870242 4168 raft_consensus.cc:493] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.870389 4168 raft_consensus.cc:515] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.871115 4168 leader_election.cc:290] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.871309 4168 raft_consensus.cc:2804] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:49.871420 4168 raft_consensus.cc:697] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:49.871825 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 86e483c52e514f1da54335479f59d68e (table default.Table_6d91747dd6164f0d84509497a93aeeea [id=f33faf56d22346bdaa270b22e673315a]) successfully deleted
I20251026 02:56:49.871727 4168 consensus_queue.cc:237] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:49.873019 1916 leader_election.cc:343] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: d43b6f22dd4f4f9c9657f21514550312
I20251026 02:56:49.873522 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d43b6f22dd4f4f9c9657f21514550312" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:49.874372 2195 consensus_peers.cc:597] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b5e61afe90e4bc4a2897de21e0b3919. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:49.876924 1873 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 06de594e17244f0d906ba296231358d4
I20251026 02:56:49.877718 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 06de594e17244f0d906ba296231358d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.878917 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c3b49514cb69429ca8b6cc1fd3422ddf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.879523 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3b49514cb69429ca8b6cc1fd3422ddf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.879727 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 886fe7ffa394440eb893887a9feaff2b
I20251026 02:56:49.880441 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1a8082b54aaf4bb4809829906a72db16" } new_table_name: "default.Table_8e0fc984298540898b1ae289f95d504f" modify_external_catalogs: true
I20251026 02:56:49.880535 1656 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_73789469dbaa4885b8566a3aa0877893" } new_table_name: "default.Table_6a6edd8b559245748a47a6539dbd2949" modify_external_catalogs: true
I20251026 02:56:49.881752 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet c95cf788ecfd4ed491ff49f9f2dc8db3
I20251026 02:56:49.882185 1655 catalog_manager.cc:6719] ReplaceTablet: tablet c95cf788ecfd4ed491ff49f9f2dc8db3 deleted and replaced by tablet ffe8d17204cb40a09992b5600958e64e
I20251026 02:56:49.882568 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a09f60f754244ce3958f5633cf37ec42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.883026 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a09f60f754244ce3958f5633cf37ec42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.884243 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 85420a102dae4269811dbdc64aebb20a
I20251026 02:56:49.884686 1657 catalog_manager.cc:6719] ReplaceTablet: tablet 85420a102dae4269811dbdc64aebb20a deleted and replaced by tablet ee46aae2e3304a3ea7ac5e02ea4b3b4b
I20251026 02:56:49.884914 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet b1b6613742a24c21a1db6ebe858a846f with delete_type TABLET_DATA_DELETED (replaced by tablet 0c60d8227eed417595ef68d44c68ebcf) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.885093 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 886fe7ffa394440eb893887a9feaff2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.885296 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet c95cf788ecfd4ed491ff49f9f2dc8db3 with delete_type TABLET_DATA_DELETED (replaced by tablet ffe8d17204cb40a09992b5600958e64e) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.885375 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fb320a7039df4723a13ea9706037a238" } new_table_name: "default.Table_e71710fddd504797843da57ccbebd51a" modify_external_catalogs: true
I20251026 02:56:49.885524 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 85420a102dae4269811dbdc64aebb20a with delete_type TABLET_DATA_DELETED (replaced by tablet ee46aae2e3304a3ea7ac5e02ea4b3b4b) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.886487 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 4984a3aea6a245ffba7f393b47add7d0
I20251026 02:56:49.887424 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 4984a3aea6a245ffba7f393b47add7d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.890010 1915 leader_election.cc:304] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:49.893249 4168 raft_consensus.cc:2749] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.894577 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 09f5265d85c6496d85c555b2b847373a (table default.Table_2bb5a8427c5044cc8dc28649dea1f4b1 [id=fe6fe74d50ea4174bc20c04bc77f8215]). Expected version 1 got 0
I20251026 02:56:49.894836 1661 catalog_manager.cc:5649] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2). New cstate: current_term: 1 leader_uuid: "0b6b0a6c4e144982b67980118cddba48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20251026 02:56:49.895244 1873 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 2386155307964833b90565b6da44ab8b
I20251026 02:56:49.896153 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 2386155307964833b90565b6da44ab8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:49.897553 4168 raft_consensus.cc:493] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.897688 4168 raft_consensus.cc:515] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.898756 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "398c1f420974453b9b6b0fe17795378b" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:49.898980 2109 raft_consensus.cc:2393] T 398c1f420974453b9b6b0fe17795378b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:49.899551 4168 leader_election.cc:290] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.902506 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "398c1f420974453b9b6b0fe17795378b" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:49.904053 4067 raft_consensus.cc:493] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.904181 4067 raft_consensus.cc:515] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.904578 2375 tablet_bootstrap.cc:492] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:49.904731 2375 ts_tablet_manager.cc:1403] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.001s
I20251026 02:56:49.904729 4067 leader_election.cc:290] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:49.905402 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 5751aed1fcf24de2bd91fc68fc1e4cd7 (DEFAULT_TABLE table=default.Table_f516b13fdea9459b84f650d89ddb6b21 [id=219e270a7b454167a9f49811b69eaa81]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:49.905582 2195 leader_election.cc:343] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 87895fb88c284a4f8d7c58a5ee48ba8e
I20251026 02:56:49.905740 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87895fb88c284a4f8d7c58a5ee48ba8e" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.905973 2375 raft_consensus.cc:359] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.906157 2375 raft_consensus.cc:385] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:49.906246 2375 raft_consensus.cc:740] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
W20251026 02:56:49.906247 2194 leader_election.cc:343] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.906491 2375 consensus_queue.cc:260] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.907063 2375 ts_tablet_manager.cc:1434] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:49.907227 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 1992d59a0b4d4e38a296f1cfb38025bf (DEFAULT_TABLE table=default.Table_355df3b4f4f94da898a80afb145a00d7 [id=60eda0030f4f423e82c8e10e566507d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.907338 2375 tablet_bootstrap.cc:492] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:49.907435 1915 leader_election.cc:304] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:49.907222 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 1c5d004de7e14fa88cb6608a97649220 (DEFAULT_TABLE table=default.Table_4464817693df4358810261706b8b254b [id=e0c800e77aa240c09a76c5260ae62f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.911394 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 06de594e17244f0d906ba296231358d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.911924 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 06de594e17244f0d906ba296231358d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.912918 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 886fe7ffa394440eb893887a9feaff2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.912794 4297 raft_consensus.cc:3060] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:49.913321 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 886fe7ffa394440eb893887a9feaff2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.913887 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet c95cf788ecfd4ed491ff49f9f2dc8db3 with delete_type TABLET_DATA_DELETED (replaced by tablet ffe8d17204cb40a09992b5600958e64e) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.914213 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet c95cf788ecfd4ed491ff49f9f2dc8db3 with delete_type TABLET_DATA_DELETED (replaced by tablet ffe8d17204cb40a09992b5600958e64e) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.914820 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet f2d97084b5914350afbfda5c2a34bf9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.915141 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 85420a102dae4269811dbdc64aebb20a with delete_type TABLET_DATA_DELETED (replaced by tablet ee46aae2e3304a3ea7ac5e02ea4b3b4b) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.915660 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 85420a102dae4269811dbdc64aebb20a with delete_type TABLET_DATA_DELETED (replaced by tablet ee46aae2e3304a3ea7ac5e02ea4b3b4b) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.916067 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 4984a3aea6a245ffba7f393b47add7d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.916414 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 4984a3aea6a245ffba7f393b47add7d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.916961 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet 2386155307964833b90565b6da44ab8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:49.895219 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet cfac07917ee34e6dba3fac5e081f6b43 (table default.Table_907991c2b0034c118065ead079f273f9 [id=a8d490f3dbc948d38c54d14e178b736e]). Expected version 1 got 0
I20251026 02:56:49.918167 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 41a1442fffe54add93666e3c78a0cc58 (table default.Table_1faa0c8151724c5e816546ee2e1d5182 [id=a106f9ece0b9467a8bc1d5d4b3524c41]). Expected version 1 got 0
I20251026 02:56:49.921394 4167 consensus_queue.cc:1048] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:49.922722 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1009) took 3390 ms. Trace:
I20251026 02:56:49.922952 2376 rpcz_store.cc:276] 1026 02:56:46.531959 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.532020 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:49.922583 (+3390563us) ts_tablet_manager.cc:1151] Deleting tablet 51a39824f5404d45b1756a2581aa1da4
1026 02:56:49.922592 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.922678 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3390369}
I20251026 02:56:49.923596 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1010) took 3338 ms. Trace:
W20251026 02:56:49.926926 3337 leader_election.cc:336] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655): Timed out: RequestConsensusVote RPC to 127.1.150.61:37655 timed out after 0.715s (SENT)
I20251026 02:56:49.927104 3337 leader_election.cc:304] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 751b719b75ce4bc2a88a19091230ecb7; no voters: 209968db4dee47c49497a66c3ee750b7, 647c0de4ce394caca75179fe11658fc8
I20251026 02:56:49.928630 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 2386155307964833b90565b6da44ab8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:49.930078 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 5751aed1fcf24de2bd91fc68fc1e4cd7 (DEFAULT_TABLE table=default.Table_f516b13fdea9459b84f650d89ddb6b21 [id=219e270a7b454167a9f49811b69eaa81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.930497 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5751aed1fcf24de2bd91fc68fc1e4cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.930915 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 5751aed1fcf24de2bd91fc68fc1e4cd7 (DEFAULT_TABLE table=default.Table_f516b13fdea9459b84f650d89ddb6b21 [id=219e270a7b454167a9f49811b69eaa81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.931365 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5751aed1fcf24de2bd91fc68fc1e4cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.931663 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 1c5d004de7e14fa88cb6608a97649220 (DEFAULT_TABLE table=default.Table_4464817693df4358810261706b8b254b [id=e0c800e77aa240c09a76c5260ae62f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.932004 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5d004de7e14fa88cb6608a97649220. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.933125 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 1992d59a0b4d4e38a296f1cfb38025bf (DEFAULT_TABLE table=default.Table_355df3b4f4f94da898a80afb145a00d7 [id=60eda0030f4f423e82c8e10e566507d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.933429 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1992d59a0b4d4e38a296f1cfb38025bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.933914 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 1c5d004de7e14fa88cb6608a97649220 (DEFAULT_TABLE table=default.Table_4464817693df4358810261706b8b254b [id=e0c800e77aa240c09a76c5260ae62f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.934197 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5d004de7e14fa88cb6608a97649220. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.944779 1946 tablet_service.cc:1505] Processing CreateTablet for tablet af3319885c99452490ede1eef9a40e2c (DEFAULT_TABLE table=default.Table_a1d0a91306ea4540bbceec66fe7abb8d [id=470640c1549040a590da43aded9bdaed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.945395 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af3319885c99452490ede1eef9a40e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.950637 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 1992d59a0b4d4e38a296f1cfb38025bf (DEFAULT_TABLE table=default.Table_355df3b4f4f94da898a80afb145a00d7 [id=60eda0030f4f423e82c8e10e566507d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.951085 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1992d59a0b4d4e38a296f1cfb38025bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.958173 2194 leader_election.cc:304] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.958544 4256 raft_consensus.cc:2749] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:49.958897 2194 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 52657.1us
I20251026 02:56:49.960520 4256 raft_consensus.cc:493] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:49.960634 4256 raft_consensus.cc:515] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.961144 4256 leader_election.cc:290] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
W20251026 02:56:49.962085 2195 leader_election.cc:343] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9c7efb70c0804c149f36440a4a9e6ef8
W20251026 02:56:49.962824 2194 consensus_peers.cc:597] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:49.963579 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c7efb70c0804c149f36440a4a9e6ef8" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.963810 1982 raft_consensus.cc:2468] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:49.966194 2242 tablet_service.cc:1505] Processing CreateTablet for tablet af3319885c99452490ede1eef9a40e2c (DEFAULT_TABLE table=default.Table_a1d0a91306ea4540bbceec66fe7abb8d [id=470640c1549040a590da43aded9bdaed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.966502 2194 leader_election.cc:304] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
W20251026 02:56:49.967662 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 51a39824f5404d45b1756a2581aa1da4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51a39824f5404d45b1756a2581aa1da4 already in progress: creating tablet
I20251026 02:56:49.968019 4255 raft_consensus.cc:2804] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 02:56:49.968089 4255 raft_consensus.cc:493] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:49.966590 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af3319885c99452490ede1eef9a40e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.968145 4255 raft_consensus.cc:3060] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Advancing to term 2
I20251026 02:56:49.975665 2081 tablet_service.cc:1505] Processing CreateTablet for tablet af3319885c99452490ede1eef9a40e2c (DEFAULT_TABLE table=default.Table_a1d0a91306ea4540bbceec66fe7abb8d [id=470640c1549040a590da43aded9bdaed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.977146 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2811eabdf1ec4f61b92ab33a8c7ec6c3" } new_table_name: "default.Table_753a441b9abc496c8c4d77321f6ddd32" modify_external_catalogs: true
I20251026 02:56:49.978942 2098 tablet_service.cc:1505] Processing CreateTablet for tablet c8412fa10d8f446787c7f8cab0ad3926 (DEFAULT_TABLE table=default.Table_62dd15084aac45f585bca051a5e7cfe5 [id=6a79c571f7834195a076d7a37db0beab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.979812 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5751aed1fcf24de2bd91fc68fc1e4cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.980095 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af3319885c99452490ede1eef9a40e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.980367 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8412fa10d8f446787c7f8cab0ad3926. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.980595 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5d004de7e14fa88cb6608a97649220. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.982928 2372 ts_tablet_manager.cc:1929] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:49.983057 2372 log.cc:1199] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/a53f25688e3b45dca68984e2c543d094
I20251026 02:56:49.983412 2372 ts_tablet_manager.cc:1950] T a53f25688e3b45dca68984e2c543d094 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:49.986429 3997 raft_consensus.cc:493] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:49.986696 3997 raft_consensus.cc:515] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:49.987476 3997 leader_election.cc:290] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:49.987960 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79c18ebf56e045b6a52fae8871a9107f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:49.988243 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1992d59a0b4d4e38a296f1cfb38025bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.988199 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79c18ebf56e045b6a52fae8871a9107f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:49.988623 2053 leader_election.cc:343] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:49.992445 2053 leader_election.cc:343] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:49.992635 2053 leader_election.cc:304] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:49.994163 3997 raft_consensus.cc:2749] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:49.996740 1944 tablet_service.cc:1505] Processing CreateTablet for tablet c8412fa10d8f446787c7f8cab0ad3926 (DEFAULT_TABLE table=default.Table_62dd15084aac45f585bca051a5e7cfe5 [id=6a79c571f7834195a076d7a37db0beab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.996987 2241 tablet_service.cc:1505] Processing CreateTablet for tablet c8412fa10d8f446787c7f8cab0ad3926 (DEFAULT_TABLE table=default.Table_62dd15084aac45f585bca051a5e7cfe5 [id=6a79c571f7834195a076d7a37db0beab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.997020 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 147a3e00fd034394b03fedf3bd7a1d86 (DEFAULT_TABLE table=default.Table_7d70886e009247ba950d471827c82385 [id=0b9e732beb0f4f2b9ae9bdf239939142]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:49.997126 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8412fa10d8f446787c7f8cab0ad3926. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.997323 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8412fa10d8f446787c7f8cab0ad3926. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.997351 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 147a3e00fd034394b03fedf3bd7a1d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:49.999033 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1ee28dad1e4148a8b092d79b5e5b2709"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:49.999660 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ee28dad1e4148a8b092d79b5e5b2709 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.000242 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 662f2f0dfc804ddbbc4b71118a71d048
I20251026 02:56:50.000648 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_73c105436cb84727b8cabb2354aae56e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.001168 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73c105436cb84727b8cabb2354aae56e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:49.985949 2375 tablet_bootstrap.cc:654] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:50.001384 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06b5ac1b5e744170b4b455f3d7af092d consensus state has no leader
W20251026 02:56:50.002190 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76c779bae091430fac46ad5dffb360ca has no consensus state
I20251026 02:56:50.003758 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 662f2f0dfc804ddbbc4b71118a71d048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.006966 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 662f2f0dfc804ddbbc4b71118a71d048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.013038 2381 tablet_bootstrap.cc:492] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:50.013307 2381 ts_tablet_manager.cc:1403] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.187s user 0.000s sys 0.007s
I20251026 02:56:50.014241 2381 raft_consensus.cc:359] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.014537 2381 raft_consensus.cc:385] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.014564 4297 raft_consensus.cc:2749] T 398c1f420974453b9b6b0fe17795378b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:50.014724 2381 raft_consensus.cc:740] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.015005 2381 consensus_queue.cc:260] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.015247 4300 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251026 02:56:50.016047 3337 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.61:37655, user_credentials={real_user=slave}} blocked reactor thread for 89153.4us
W20251026 02:56:50.017827 1916 consensus_peers.cc:597] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cf76ef8f6af445eb0ce15c8d79136c5. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:50.018168 1916 consensus_peers.cc:597] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a345fa5edbd48d2891b8b40b09cb3f8. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:50.018932 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 662f2f0dfc804ddbbc4b71118a71d048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:49 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.019140 2378 tablet_bootstrap.cc:492] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:50.019564 2378 ts_tablet_manager.cc:1403] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.239s user 0.005s sys 0.005s
I20251026 02:56:50.019908 4265 tablet.cc:1722] T 94886ad6d6bb45f39261725169774ac9 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:50.020597 2375 tablet_bootstrap.cc:492] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:50.021247 4266 tablet.cc:1722] T 94886ad6d6bb45f39261725169774ac9 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:50.021752 2375 ts_tablet_manager.cc:1403] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.114s user 0.002s sys 0.002s
I20251026 02:56:50.022576 2375 raft_consensus.cc:359] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.022838 2375 raft_consensus.cc:385] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.022939 2375 raft_consensus.cc:740] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.023226 2375 consensus_queue.cc:260] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.023870 2375 ts_tablet_manager.cc:1434] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 02:56:50.024204 2375 tablet_bootstrap.cc:492] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:50.026597 2378 raft_consensus.cc:359] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.027542 2378 raft_consensus.cc:385] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.027805 2378 raft_consensus.cc:740] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.027853 2377 ts_tablet_manager.cc:1929] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.028060 2377 log.cc:1199] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/f719c720cc474181a63786a3e248ab2c
I20251026 02:56:50.028321 2378 consensus_queue.cc:260] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.028620 2377 ts_tablet_manager.cc:1950] T f719c720cc474181a63786a3e248ab2c P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.032754 4271 tablet.cc:1722] T 94886ad6d6bb45f39261725169774ac9 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251026 02:56:50.037722 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 14aac5b9c1de47b496ebe7ea4cd7a766 (DEFAULT_TABLE table=default.Table_e0fa37c306924b8ea7a9cd90c7d94b38 [id=60ff199aaa844a2991e1ca206f993a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.038812 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 147a3e00fd034394b03fedf3bd7a1d86 (DEFAULT_TABLE table=default.Table_7d70886e009247ba950d471827c82385 [id=0b9e732beb0f4f2b9ae9bdf239939142]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.038903 2378 ts_tablet_manager.cc:1434] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251026 02:56:50.039778 2378 tablet_bootstrap.cc:492] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:50.029131 4255 raft_consensus.cc:515] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.040402 4255 leader_election.cc:290] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:50.040679 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14aac5b9c1de47b496ebe7ea4cd7a766. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.041182 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c7efb70c0804c149f36440a4a9e6ef8" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:50.041451 1975 raft_consensus.cc:3060] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 2
I20251026 02:56:50.041872 2378 tablet_bootstrap.cc:654] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.043464 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f28a7b90302c4e3993516e647d2bc67c" } modify_external_catalogs: true
I20251026 02:56:50.043678 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f28a7b90302c4e3993516e647d2bc67c" } modify_external_catalogs: true
I20251026 02:56:50.044312 2084 tablet_service.cc:1505] Processing CreateTablet for tablet da49ae245eb84f5793a0e765b4b172cd (DEFAULT_TABLE table=default.Table_c29fa2ff53174902a60e57cb90cb11c8 [id=596614d8188b4787891466b58166c35e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.044459 1975 raft_consensus.cc:2468] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 2.
I20251026 02:56:50.044675 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da49ae245eb84f5793a0e765b4b172cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.044869 2194 leader_election.cc:304] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:50.045147 4255 raft_consensus.cc:2804] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Leader election won for term 2
W20251026 02:56:50.045219 2195 leader_election.cc:343] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9c7efb70c0804c149f36440a4a9e6ef8
I20251026 02:56:50.045279 4255 raft_consensus.cc:697] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 2 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:50.045507 4255 consensus_queue.cc:237] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.046371 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_45834367050843509698000b4af518a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.047120 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45834367050843509698000b4af518a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.048482 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 147a3e00fd034394b03fedf3bd7a1d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.049546 1662 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_79ee525e9c034edbb93a0f58159e20ce" } new_table_name: "default.Table_eb5ffb759fdb4711a6174118713e2180" modify_external_catalogs: true
I20251026 02:56:50.051803 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_cbbe03e029b7424d99157ea61f5a01ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.052524 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbbe03e029b7424d99157ea61f5a01ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.055452 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 976) took 6170 ms. Trace:
I20251026 02:56:50.055574 2377 rpcz_store.cc:276] 1026 02:56:43.885320 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:43.885371 (+ 51us) service_pool.cc:225] Handling call
1026 02:56:49.859832 (+5974461us) ts_tablet_manager.cc:1151] Deleting tablet f719c720cc474181a63786a3e248ab2c
1026 02:56:49.859847 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.894619 (+ 34772us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.027729 (+133110us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.055429 (+ 27700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5974241}
I20251026 02:56:50.055968 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 981) took 5903 ms. Trace:
I20251026 02:56:50.056051 2377 rpcz_store.cc:276] 1026 02:56:44.152500 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.152551 (+ 51us) service_pool.cc:225] Handling call
1026 02:56:50.055844 (+5903293us) ts_tablet_manager.cc:1151] Deleting tablet 0afb6f04f5ac463780fe8f522bf2fe23
1026 02:56:50.055854 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.055952 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5903092}
I20251026 02:56:50.056380 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 983) took 5887 ms. Trace:
I20251026 02:56:50.039638 2381 ts_tablet_manager.cc:1434] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.026s user 0.000s sys 0.001s
I20251026 02:56:50.056872 2381 tablet_bootstrap.cc:492] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:50.058246 2381 tablet_bootstrap.cc:654] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.058952 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 2f5cd2c13b034500bcbc655680999ca0 (DEFAULT_TABLE table=default.Table_c6919d91bdbd4df3965482f54a8c703e [id=fa2b008189f84228bf535de06b59501e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.059310 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f5cd2c13b034500bcbc655680999ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.059252 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 14aac5b9c1de47b496ebe7ea4cd7a766 (DEFAULT_TABLE table=default.Table_e0fa37c306924b8ea7a9cd90c7d94b38 [id=60ff199aaa844a2991e1ca206f993a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.059399 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 2f5cd2c13b034500bcbc655680999ca0 (DEFAULT_TABLE table=default.Table_c6919d91bdbd4df3965482f54a8c703e [id=fa2b008189f84228bf535de06b59501e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.059602 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14aac5b9c1de47b496ebe7ea4cd7a766. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.059824 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f5cd2c13b034500bcbc655680999ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.060353 1943 tablet_service.cc:1505] Processing CreateTablet for tablet da49ae245eb84f5793a0e765b4b172cd (DEFAULT_TABLE table=default.Table_c29fa2ff53174902a60e57cb90cb11c8 [id=596614d8188b4787891466b58166c35e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.060639 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da49ae245eb84f5793a0e765b4b172cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.064560 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 14aac5b9c1de47b496ebe7ea4cd7a766 (DEFAULT_TABLE table=default.Table_e0fa37c306924b8ea7a9cd90c7d94b38 [id=60ff199aaa844a2991e1ca206f993a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.064939 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14aac5b9c1de47b496ebe7ea4cd7a766. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.067181 2235 tablet_service.cc:1505] Processing CreateTablet for tablet da49ae245eb84f5793a0e765b4b172cd (DEFAULT_TABLE table=default.Table_c29fa2ff53174902a60e57cb90cb11c8 [id=596614d8188b4787891466b58166c35e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.067474 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da49ae245eb84f5793a0e765b4b172cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.067519 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 147a3e00fd034394b03fedf3bd7a1d86 (DEFAULT_TABLE table=default.Table_7d70886e009247ba950d471827c82385 [id=0b9e732beb0f4f2b9ae9bdf239939142]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.067970 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 8ed7a9f510dd4297b581bd0c4f610b48 (DEFAULT_TABLE table=default.Table_f1f3e2329722434cae69c91816fab475 [id=0cedb08dd0064272a185d62c7641f12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.068285 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed7a9f510dd4297b581bd0c4f610b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.068527 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 147a3e00fd034394b03fedf3bd7a1d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.077608 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 2f5cd2c13b034500bcbc655680999ca0 (DEFAULT_TABLE table=default.Table_c6919d91bdbd4df3965482f54a8c703e [id=fa2b008189f84228bf535de06b59501e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.078094 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f5cd2c13b034500bcbc655680999ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.056448 2377 rpcz_store.cc:276] 1026 02:56:44.168417 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.168565 (+ 148us) service_pool.cc:225] Handling call
1026 02:56:50.056284 (+5887719us) ts_tablet_manager.cc:1151] Deleting tablet 0c60d8227eed417595ef68d44c68ebcf
1026 02:56:50.056292 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.056366 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5887535}
I20251026 02:56:50.079836 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 982) took 5911 ms. Trace:
W20251026 02:56:50.086478 1918 connection.cc:537] client connection to 127.1.150.61:37655 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251026 02:56:50.086699 3337 connection.cc:537] client connection to 127.1.150.61:37655 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251026 02:56:50.087051 3335 connection.cc:537] server connection from 127.0.0.1:57038 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251026 02:56:50.087162 2056 connection.cc:537] client connection to 127.1.150.61:37655 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251026 02:56:50.088403 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1039) took 4583 ms. Trace:
W20251026 02:56:50.089022 1650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.150.61:37655 (error 108) [suppressed 10 similar messages]
I20251026 02:56:50.089936 1657 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet cfac07917ee34e6dba3fac5e081f6b43 (table default.Table_907991c2b0034c118065ead079f273f9 [id=a8d490f3dbc948d38c54d14e178b736e]). Expected version 1 got 0
I20251026 02:56:50.090315 1657 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 398c1f420974453b9b6b0fe17795378b (table default.Table_a92e68d05f4b455d810180ee4cc5874d [id=7f64fbec3582415a8924d01e14856a56]). Expected version 1 got 0
W20251026 02:56:50.087430 2197 connection.cc:537] client connection to 127.1.150.61:37655 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251026 02:56:50.094422 2377 rpcz_store.cc:276] 1026 02:56:44.168274 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.168326 (+ 52us) service_pool.cc:225] Handling call
1026 02:56:50.079711 (+5911385us) ts_tablet_manager.cc:1151] Deleting tablet 146d6f13b9ef47f29711a102791bab5a
1026 02:56:50.079721 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.079816 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5910913}
I20251026 02:56:50.095762 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 984) took 5924 ms. Trace:
I20251026 02:56:50.095849 2377 rpcz_store.cc:276] 1026 02:56:44.171454 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.171508 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:50.095611 (+5924103us) ts_tablet_manager.cc:1151] Deleting tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1
1026 02:56:50.095624 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.095746 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5923933}
I20251026 02:56:50.096686 2372 rpcz_store.cc:276] 1026 02:56:45.504534 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.504874 (+ 340us) service_pool.cc:225] Handling call
1026 02:56:49.858608 (+4353734us) ts_tablet_manager.cc:1151] Deleting tablet a53f25688e3b45dca68984e2c543d094
1026 02:56:49.858616 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.866377 (+ 7761us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:49.982797 (+116420us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.088364 (+105567us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3531,"tablet-delete.queue_time_us":4350206}
I20251026 02:56:50.097666 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 985) took 5926 ms. Trace:
I20251026 02:56:50.097779 2377 rpcz_store.cc:276] 1026 02:56:44.171594 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.171703 (+ 109us) service_pool.cc:225] Handling call
1026 02:56:50.097529 (+5925826us) ts_tablet_manager.cc:1151] Deleting tablet 63f1abc4477f4198ad2cc8481d69df73
1026 02:56:50.097550 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.097649 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5925706}
I20251026 02:56:50.098330 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 986) took 5923 ms. Trace:
I20251026 02:56:50.098428 2377 rpcz_store.cc:276] 1026 02:56:44.175295 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.176050 (+ 755us) service_pool.cc:225] Handling call
1026 02:56:50.098197 (+5922147us) ts_tablet_manager.cc:1151] Deleting tablet 5eda85f2d53a43d68907c467c7da9bd6
1026 02:56:50.098209 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.098314 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5921979}
I20251026 02:56:50.098968 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1040) took 4584 ms. Trace:
I20251026 02:56:50.099109 2372 rpcz_store.cc:276] 1026 02:56:45.514190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.514272 (+ 82us) service_pool.cc:225] Handling call
1026 02:56:50.097047 (+4582775us) ts_tablet_manager.cc:1151] Deleting tablet f437d8b5f5784e8fbec10d0700e90348
1026 02:56:50.098318 (+ 1271us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.098935 (+ 617us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4582465}
I20251026 02:56:50.099325 4256 raft_consensus.cc:493] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.099525 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1037) took 4598 ms. Trace:
I20251026 02:56:50.099449 4256 raft_consensus.cc:515] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.099627 2372 rpcz_store.cc:276] 1026 02:56:45.501549 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.550226 (+ 48677us) service_pool.cc:225] Handling call
1026 02:56:50.099394 (+4549168us) ts_tablet_manager.cc:1151] Deleting tablet 4e1291f47f384ff38eadd16040db65b8
1026 02:56:50.099404 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.099487 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4545900}
I20251026 02:56:50.099506 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 987) took 5871 ms. Trace:
I20251026 02:56:50.099931 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1042) took 4333 ms. Trace:
I20251026 02:56:50.100081 4256 leader_election.cc:290] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:50.100044 2372 rpcz_store.cc:276] 1026 02:56:45.766540 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.766626 (+ 86us) service_pool.cc:225] Handling call
1026 02:56:50.099823 (+4333197us) ts_tablet_manager.cc:1151] Deleting tablet 53264ef08ff7454faa19b18fa35bd803
1026 02:56:50.099831 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.099902 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4322522}
I20251026 02:56:50.099844 2377 rpcz_store.cc:276] 1026 02:56:44.227775 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.272357 (+ 44582us) service_pool.cc:225] Handling call
1026 02:56:50.099391 (+5827034us) ts_tablet_manager.cc:1151] Deleting tablet 146d6f13b9ef47f29711a102791bab5a
1026 02:56:50.099415 (+ 24us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.099491 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5826811}
I20251026 02:56:50.100490 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b77bb6b4722449c9699b548a9ef6314" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:50.100540 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b77bb6b4722449c9699b548a9ef6314" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:50.100689 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1000) took 5355 ms. Trace:
I20251026 02:56:50.100747 2109 raft_consensus.cc:2468] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:50.100749 2377 rpcz_store.cc:276] 1026 02:56:44.744800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:44.748312 (+ 3512us) service_pool.cc:225] Handling call
1026 02:56:50.100599 (+5352287us) ts_tablet_manager.cc:1151] Deleting tablet e45600a92f8e46b096badd700779856d
1026 02:56:50.100608 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.100676 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5352075}
I20251026 02:56:50.101150 2195 leader_election.cc:304] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
W20251026 02:56:50.101367 2194 leader_election.cc:343] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.101629 4258 raft_consensus.cc:2804] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:50.101701 4258 raft_consensus.cc:493] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:50.101768 4258 raft_consensus.cc:3060] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.101850 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1007) took 5095 ms. Trace:
I20251026 02:56:50.101908 2377 rpcz_store.cc:276] 1026 02:56:45.006121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.006208 (+ 87us) service_pool.cc:225] Handling call
1026 02:56:50.101018 (+5094810us) ts_tablet_manager.cc:1151] Deleting tablet d90e62ae86b149b291dc29cf7442c6bc
1026 02:56:50.101026 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.101837 (+ 811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5094501}
I20251026 02:56:50.102275 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1008) took 5092 ms. Trace:
I20251026 02:56:50.102337 2377 rpcz_store.cc:276] 1026 02:56:45.009657 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.009720 (+ 63us) service_pool.cc:225] Handling call
1026 02:56:50.102201 (+5092481us) ts_tablet_manager.cc:1151] Deleting tablet 5c6e3e47b3e7421e945cfbe671b06abe
1026 02:56:50.102209 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.102263 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5092262}
I20251026 02:56:50.102790 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1033) took 4755 ms. Trace:
I20251026 02:56:50.102846 2377 rpcz_store.cc:276] 1026 02:56:45.347202 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.347271 (+ 69us) service_pool.cc:225] Handling call
1026 02:56:50.102710 (+4755439us) ts_tablet_manager.cc:1151] Deleting tablet 6ce4092b7ccc4888b5cbe3b38bee2e36
1026 02:56:50.102718 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.102779 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4755153}
I20251026 02:56:50.103178 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1032) took 4764 ms. Trace:
I20251026 02:56:50.103228 2377 rpcz_store.cc:276] 1026 02:56:45.338976 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.350782 (+ 11806us) service_pool.cc:225] Handling call
1026 02:56:50.103090 (+4752308us) ts_tablet_manager.cc:1151] Deleting tablet 3865399cf6934bc18e16ae2cb4887197
1026 02:56:50.103097 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.103155 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4752097}
I20251026 02:56:50.103595 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1034) took 4710 ms. Trace:
I20251026 02:56:50.103716 2377 rpcz_store.cc:276] 1026 02:56:45.393285 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.403668 (+ 10383us) service_pool.cc:225] Handling call
1026 02:56:50.103501 (+4699833us) ts_tablet_manager.cc:1151] Deleting tablet 4e1291f47f384ff38eadd16040db65b8
1026 02:56:50.103509 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.103581 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4699557}
I20251026 02:56:50.104035 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1035) took 4705 ms. Trace:
I20251026 02:56:50.104099 2377 rpcz_store.cc:276] 1026 02:56:45.447082 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.447118 (+ 36us) service_pool.cc:225] Handling call
1026 02:56:50.103958 (+4656840us) ts_tablet_manager.cc:1151] Deleting tablet 94d5faa4584a4e36b38bdd552d641994
1026 02:56:50.103966 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.104024 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4656620}
I20251026 02:56:50.104493 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1039) took 4595 ms. Trace:
I20251026 02:56:50.104624 2377 rpcz_store.cc:276] 1026 02:56:45.509210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.509270 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:50.104430 (+4595160us) ts_tablet_manager.cc:1151] Deleting tablet 7801f073d5d94076af9299492f86fad2
1026 02:56:50.104437 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.104482 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4594943}
I20251026 02:56:50.104686 4258 raft_consensus.cc:515] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.105013 2377 tablet_replica.cc:333] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.105235 4258 leader_election.cc:290] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:50.105533 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 41a1442fffe54add93666e3c78a0cc58 (table default.Table_1faa0c8151724c5e816546ee2e1d5182 [id=a106f9ece0b9467a8bc1d5d4b3524c41]). Expected version 1 got 0
I20251026 02:56:50.105676 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b77bb6b4722449c9699b548a9ef6314" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:50.105710 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b77bb6b4722449c9699b548a9ef6314" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:50.105908 2109 raft_consensus.cc:3060] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:50.106230 2194 leader_election.cc:343] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.106465 2377 raft_consensus.cc:2243] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:50.106873 2377 raft_consensus.cc:2272] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.107542 2377 ts_tablet_manager.cc:1916] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.108881 2109 raft_consensus.cc:2468] T 3b77bb6b4722449c9699b548a9ef6314 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:50.109791 2195 leader_election.cc:304] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:50.110316 4258 raft_consensus.cc:2804] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:50.110481 4258 raft_consensus.cc:697] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:50.110759 4258 consensus_queue.cc:237] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.112617 2375 tablet_bootstrap.cc:654] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.114547 2372 tablet_replica.cc:333] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:50.115568 2372 raft_consensus.cc:2243] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:50.115744 2372 raft_consensus.cc:2272] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:50.117398 1629 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 49749.9us
I20251026 02:56:50.119161 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet f719c720cc474181a63786a3e248ab2c (table default.Table_e586d81375544236ac7d95297fca01a8 [id=67731996dfc04572b0dfebf0edb71942]) successfully deleted
W20251026 02:56:50.119665 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0afb6f04f5ac463780fe8f522bf2fe23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0afb6f04f5ac463780fe8f522bf2fe23 already in progress: creating tablet
W20251026 02:56:50.120126 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0c60d8227eed417595ef68d44c68ebcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c60d8227eed417595ef68d44c68ebcf already in progress: creating tablet
I20251026 02:56:50.123790 2375 tablet_bootstrap.cc:492] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:50.123878 2377 ts_tablet_manager.cc:1929] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.123944 2375 ts_tablet_manager.cc:1403] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.000s
I20251026 02:56:50.124078 2377 log.cc:1199] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/34c6a7c774044411a4733b115080c4c6
I20251026 02:56:50.124471 2377 ts_tablet_manager.cc:1950] T 34c6a7c774044411a4733b115080c4c6 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.124732 2375 raft_consensus.cc:359] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.124907 2375 raft_consensus.cc:385] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.124982 2375 raft_consensus.cc:740] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.125404 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1040) took 4616 ms. Trace:
I20251026 02:56:50.125564 2377 rpcz_store.cc:276] 1026 02:56:45.509329 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.509527 (+ 198us) service_pool.cc:225] Handling call
1026 02:56:50.104976 (+4595449us) ts_tablet_manager.cc:1151] Deleting tablet 34c6a7c774044411a4733b115080c4c6
1026 02:56:50.104984 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.114250 (+ 9266us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.123773 (+ 9523us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.125385 (+ 1612us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4595321}
I20251026 02:56:50.126070 2377 tablet_replica.cc:333] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.126317 2377 raft_consensus.cc:2243] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:50.126420 2372 ts_tablet_manager.cc:1916] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.126756 2377 raft_consensus.cc:2272] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.127338 2377 ts_tablet_manager.cc:1916] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.129848 2378 tablet_bootstrap.cc:492] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:50.129995 2378 ts_tablet_manager.cc:1403] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.000s
I20251026 02:56:50.130779 2378 raft_consensus.cc:359] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.130939 2378 raft_consensus.cc:385] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.130990 2378 raft_consensus.cc:740] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
W20251026 02:56:50.131929 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 146d6f13b9ef47f29711a102791bab5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 146d6f13b9ef47f29711a102791bab5a already in progress: creating tablet
W20251026 02:56:50.132181 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a6a66a9385a47e1bc20dd21bc9ec2b1 already in progress: creating tablet
W20251026 02:56:50.132462 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 63f1abc4477f4198ad2cc8481d69df73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63f1abc4477f4198ad2cc8481d69df73 already in progress: creating tablet
W20251026 02:56:50.132678 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 5eda85f2d53a43d68907c467c7da9bd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5eda85f2d53a43d68907c467c7da9bd6 already in progress: creating tablet
W20251026 02:56:50.132915 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 146d6f13b9ef47f29711a102791bab5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 146d6f13b9ef47f29711a102791bab5a already in progress: creating tablet
I20251026 02:56:50.136094 2377 ts_tablet_manager.cc:1929] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.136265 2377 log.cc:1199] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/a012d868a78f4324a92db83c81931b9d
I20251026 02:56:50.136740 2377 ts_tablet_manager.cc:1950] T a012d868a78f4324a92db83c81931b9d P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.136989 2378 consensus_queue.cc:260] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.137526 2378 ts_tablet_manager.cc:1434] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251026 02:56:50.137583 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1037) took 4628 ms. Trace:
I20251026 02:56:50.137805 2378 tablet_bootstrap.cc:492] T a53f25688e3b45dca68984e2c543d094 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:50.137737 2377 rpcz_store.cc:276] 1026 02:56:45.508966 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.512407 (+ 3441us) service_pool.cc:225] Handling call
1026 02:56:50.126021 (+4613614us) ts_tablet_manager.cc:1151] Deleting tablet a012d868a78f4324a92db83c81931b9d
1026 02:56:50.126032 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.133028 (+ 6996us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.136009 (+ 2981us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.137562 (+ 1553us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4612749}
I20251026 02:56:50.138355 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1036) took 4629 ms. Trace:
I20251026 02:56:50.138494 2377 rpcz_store.cc:276] 1026 02:56:45.508844 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.513345 (+ 4501us) service_pool.cc:225] Handling call
1026 02:56:50.138257 (+4624912us) ts_tablet_manager.cc:1151] Deleting tablet 3b7d85a2941545ceaa16b22e8328b4ff
1026 02:56:50.138267 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.138337 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4624463}
I20251026 02:56:50.138971 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1038) took 4629 ms. Trace:
I20251026 02:56:50.139092 2377 rpcz_store.cc:276] 1026 02:56:45.509097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.513025 (+ 3928us) service_pool.cc:225] Handling call
1026 02:56:50.138899 (+4625874us) ts_tablet_manager.cc:1151] Deleting tablet e701d4023fee45d18c48997aab3227c2
1026 02:56:50.138906 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.138957 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4624953}
I20251026 02:56:50.139328 2378 tablet_bootstrap.cc:654] T a53f25688e3b45dca68984e2c543d094 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.139698 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1042) took 4354 ms. Trace:
I20251026 02:56:50.139807 2377 rpcz_store.cc:276] 1026 02:56:45.785278 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.785351 (+ 73us) service_pool.cc:225] Handling call
1026 02:56:50.139597 (+4354246us) ts_tablet_manager.cc:1151] Deleting tablet 53264ef08ff7454faa19b18fa35bd803
1026 02:56:50.139605 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.139682 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4354055}
I20251026 02:56:50.140322 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1044) took 4354 ms. Trace:
I20251026 02:56:50.140431 2377 rpcz_store.cc:276] 1026 02:56:45.785493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.785667 (+ 174us) service_pool.cc:225] Handling call
1026 02:56:50.140197 (+4354530us) ts_tablet_manager.cc:1151] Deleting tablet 885c10070d6942f7abe52c4558d887e6
1026 02:56:50.140236 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.140307 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4354400}
W20251026 02:56:50.140604 1650 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.61:37655, user_credentials={real_user=slave}} blocked reactor thread for 51602.2us
I20251026 02:56:50.140902 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1045) took 4202 ms. Trace:
I20251026 02:56:50.140975 2375 consensus_queue.cc:260] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.141018 2377 rpcz_store.cc:276] 1026 02:56:45.938591 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.938654 (+ 63us) service_pool.cc:225] Handling call
1026 02:56:50.140811 (+4202157us) ts_tablet_manager.cc:1151] Deleting tablet 9b976c87b5d04f9294add7761a2afbfd
1026 02:56:50.140818 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.140887 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4201954}
I20251026 02:56:50.141597 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1046) took 4092 ms. Trace:
I20251026 02:56:50.141707 2377 rpcz_store.cc:276] 1026 02:56:46.048787 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.048847 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:50.141496 (+4092649us) ts_tablet_manager.cc:1151] Deleting tablet 6a3491b303f94fe2a2c838297f73d464
1026 02:56:50.141504 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.141579 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4092446}
I20251026 02:56:50.141948 2381 tablet_bootstrap.cc:492] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:50.142088 2381 ts_tablet_manager.cc:1403] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.085s user 0.002s sys 0.002s
I20251026 02:56:50.142236 2377 tablet_replica.cc:333] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.142462 2377 raft_consensus.cc:2243] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:50.142657 2377 raft_consensus.cc:2272] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.142848 2381 raft_consensus.cc:359] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.143004 2381 raft_consensus.cc:385] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.143061 2381 raft_consensus.cc:740] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.143231 2381 consensus_queue.cc:260] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.143733 2381 ts_tablet_manager.cc:1434] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:50.144181 2377 ts_tablet_manager.cc:1916] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:50.144413 1650 consensus_peers.cc:597] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 -> Peer 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655): Couldn't send request to peer 647c0de4ce394caca75179fe11658fc8. Status: Network error: Client connection negotiation failed: client connection to 127.1.150.61:37655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:50.150672 2375 ts_tablet_manager.cc:1434] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20251026 02:56:50.150697 2378 tablet_bootstrap.cc:492] T a53f25688e3b45dca68984e2c543d094 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:50.150844 2378 ts_tablet_manager.cc:1403] T a53f25688e3b45dca68984e2c543d094 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20251026 02:56:50.151026 2375 tablet_bootstrap.cc:492] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:50.151538 2378 raft_consensus.cc:359] T a53f25688e3b45dca68984e2c543d094 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.151703 2378 raft_consensus.cc:385] T a53f25688e3b45dca68984e2c543d094 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.151753 2378 raft_consensus.cc:740] T a53f25688e3b45dca68984e2c543d094 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.152783 2375 tablet_bootstrap.cc:654] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.156565 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet a53f25688e3b45dca68984e2c543d094 (table default.Table_feb822201306444ebe478734d286a667 [id=535d3143b66249a083853a11934d994e]) successfully deleted
W20251026 02:56:50.156987 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet f437d8b5f5784e8fbec10d0700e90348 because the tablet was not found. No further retry: Not found: Tablet not found: f437d8b5f5784e8fbec10d0700e90348
W20251026 02:56:50.157327 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 4e1291f47f384ff38eadd16040db65b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e1291f47f384ff38eadd16040db65b8 already in progress: creating tablet
I20251026 02:56:50.158625 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 8ed7a9f510dd4297b581bd0c4f610b48 (DEFAULT_TABLE table=default.Table_f1f3e2329722434cae69c91816fab475 [id=0cedb08dd0064272a185d62c7641f12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.159135 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed7a9f510dd4297b581bd0c4f610b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.159638 2381 tablet_bootstrap.cc:492] T cfac07917ee34e6dba3fac5e081f6b43 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:50.161026 1916 consensus_peers.cc:597] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2d6691bb4cf4a8292b0c082e54a91a5. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:50.161556 2381 tablet_bootstrap.cc:654] T cfac07917ee34e6dba3fac5e081f6b43 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.163772 2378 consensus_queue.cc:260] T a53f25688e3b45dca68984e2c543d094 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.164317 2378 ts_tablet_manager.cc:1434] T a53f25688e3b45dca68984e2c543d094 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251026 02:56:50.164602 2378 tablet_bootstrap.cc:492] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:50.164548 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 04fae9383a7f45d299b11af554d09df2 (DEFAULT_TABLE table=default.Table_deeb6a15a5424e609e622a6c71b3553a [id=d0d4f71c191b41f8a4240c2fae4d28fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.164727 2377 ts_tablet_manager.cc:1929] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.164862 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04fae9383a7f45d299b11af554d09df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.164871 2377 log.cc:1199] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/62d30d79a6e4432a8dbe40a104e87fa0
I20251026 02:56:50.165262 2377 ts_tablet_manager.cc:1950] T 62d30d79a6e4432a8dbe40a104e87fa0 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.165928 2378 tablet_bootstrap.cc:654] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.166047 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1047) took 4117 ms. Trace:
I20251026 02:56:50.166153 2377 rpcz_store.cc:276] 1026 02:56:46.048908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.049080 (+ 172us) service_pool.cc:225] Handling call
1026 02:56:50.142083 (+4093003us) ts_tablet_manager.cc:1151] Deleting tablet 62d30d79a6e4432a8dbe40a104e87fa0
1026 02:56:50.142204 (+ 121us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.147179 (+ 4975us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.164583 (+ 17404us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.166026 (+ 1443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4092862}
I20251026 02:56:50.166733 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1055) took 4000 ms. Trace:
I20251026 02:56:50.166857 2377 rpcz_store.cc:276] 1026 02:56:46.166627 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.166688 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:50.166616 (+3999928us) ts_tablet_manager.cc:1151] Deleting tablet 142043b5d558490c9399626c3764ff7e
1026 02:56:50.166624 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.166715 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3999762}
I20251026 02:56:50.167315 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1056) took 4000 ms. Trace:
I20251026 02:56:50.167618 2377 rpcz_store.cc:276] 1026 02:56:46.166733 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.167003 (+ 270us) service_pool.cc:225] Handling call
1026 02:56:50.167217 (+4000214us) ts_tablet_manager.cc:1151] Deleting tablet 586bd6e40a7a4785b71b2a3d542e2411
1026 02:56:50.167226 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.167301 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4000066}
I20251026 02:56:50.168202 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1054) took 4001 ms. Trace:
I20251026 02:56:50.168555 2377 rpcz_store.cc:276] 1026 02:56:46.166519 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.167185 (+ 666us) service_pool.cc:225] Handling call
1026 02:56:50.168115 (+4000930us) ts_tablet_manager.cc:1151] Deleting tablet 075f6ca8787943a09443863c544a1284
1026 02:56:50.168122 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.168188 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4000807}
I20251026 02:56:50.169131 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1063) took 3910 ms. Trace:
I20251026 02:56:50.169450 2377 rpcz_store.cc:276] 1026 02:56:46.259079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.259925 (+ 846us) service_pool.cc:225] Handling call
1026 02:56:50.169045 (+3909120us) ts_tablet_manager.cc:1151] Deleting tablet 7bef60e3a10843b4be2022ab4b64be68
1026 02:56:50.169053 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.169119 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3908971}
I20251026 02:56:50.170090 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1064) took 3910 ms. Trace:
I20251026 02:56:50.170389 2377 rpcz_store.cc:276] 1026 02:56:46.259231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.259278 (+ 47us) service_pool.cc:225] Handling call
1026 02:56:50.170005 (+3910727us) ts_tablet_manager.cc:1151] Deleting tablet e4c149e8b7f9420c8dba56da2e21f3f9
1026 02:56:50.170013 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.170077 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3909797}
I20251026 02:56:50.170907 2377 tablet_replica.cc:333] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.171245 2377 raft_consensus.cc:2243] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:50.171852 2377 raft_consensus.cc:2272] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.172559 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 7ab71730fb5e44589046a6ff0cb516b7 (DEFAULT_TABLE table=default.Table_e5f896de4e0d426da281ce2ec2724a04 [id=d801ee46b9f442c0a22fa4b142877a98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.172915 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab71730fb5e44589046a6ff0cb516b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.173269 2377 ts_tablet_manager.cc:1916] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.176697 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 41cd60fe93d44cc4b07789a6288d86d9 (DEFAULT_TABLE table=default.Table_0b10d09d420e46ac9a13bada7fd2f740 [id=0c49b29c6a764cbe91959c5909e09496]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.177047 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41cd60fe93d44cc4b07789a6288d86d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.179558 2377 ts_tablet_manager.cc:1929] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.179641 2377 log.cc:1199] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/8c1e63ac8c5645f3ba1578792c0a0dae
I20251026 02:56:50.179939 2377 ts_tablet_manager.cc:1950] T 8c1e63ac8c5645f3ba1578792c0a0dae P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.185925 2378 tablet_bootstrap.cc:492] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:50.187395 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1065) took 3764 ms. Trace:
I20251026 02:56:50.187980 1982 raft_consensus.cc:3060] T a53f25688e3b45dca68984e2c543d094 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.188081 2377 rpcz_store.cc:276] 1026 02:56:46.423227 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.442596 (+ 19369us) service_pool.cc:225] Handling call
1026 02:56:50.170867 (+3728271us) ts_tablet_manager.cc:1151] Deleting tablet 8c1e63ac8c5645f3ba1578792c0a0dae
1026 02:56:50.170875 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.176551 (+ 5676us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.179471 (+ 2920us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.187375 (+ 7904us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3727405}
I20251026 02:56:50.188963 2377 tablet_replica.cc:333] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.189149 2377 raft_consensus.cc:2243] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:50.189294 2377 raft_consensus.cc:2272] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.189814 2377 ts_tablet_manager.cc:1916] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.190266 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 1e9482cdd69647dab6bcc2e1c75e4650 (DEFAULT_TABLE table=default.Table_1eca935882b04878bb1362a022e82ecf [id=732d51d0d5174293985c760d55f6f46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.190640 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e9482cdd69647dab6bcc2e1c75e4650. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.194193 4256 consensus_queue.cc:1048] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:50.198424 2378 ts_tablet_manager.cc:1403] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20251026 02:56:50.199620 2378 raft_consensus.cc:359] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.200984 2378 raft_consensus.cc:385] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.201241 2378 raft_consensus.cc:740] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.201828 2372 ts_tablet_manager.cc:1929] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:50.201956 2372 log.cc:1199] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/885c10070d6942f7abe52c4558d887e6
I20251026 02:56:50.202289 2372 ts_tablet_manager.cc:1950] T 885c10070d6942f7abe52c4558d887e6 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:50.201776 2378 consensus_queue.cc:260] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.203128 2378 ts_tablet_manager.cc:1434] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:50.203219 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1043) took 4427 ms. Trace:
I20251026 02:56:50.203380 2377 ts_tablet_manager.cc:1929] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.203302 2372 rpcz_store.cc:276] 1026 02:56:45.775394 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.775462 (+ 68us) service_pool.cc:225] Handling call
1026 02:56:50.114461 (+4338999us) ts_tablet_manager.cc:1151] Deleting tablet 885c10070d6942f7abe52c4558d887e6
1026 02:56:50.114471 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.129726 (+ 15255us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.201699 (+ 71973us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.203199 (+ 1500us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":777,"tablet-delete.queue_time_us":4326523}
I20251026 02:56:50.203729 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1044) took 4266 ms. Trace:
I20251026 02:56:50.203805 2372 rpcz_store.cc:276] 1026 02:56:45.937077 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:45.937147 (+ 70us) service_pool.cc:225] Handling call
1026 02:56:50.203593 (+4266446us) ts_tablet_manager.cc:1151] Deleting tablet 9b976c87b5d04f9294add7761a2afbfd
1026 02:56:50.203604 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.203714 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4197149}
I20251026 02:56:50.203964 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 885c10070d6942f7abe52c4558d887e6 (table default.Table_0982fce6dbaa41f69b6e80e254f245db [id=39a1785e03fd45b7a8ea0f14bf5a362a]) successfully deleted
I20251026 02:56:50.204247 2378 tablet_bootstrap.cc:492] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:50.204404 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 9b976c87b5d04f9294add7761a2afbfd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b976c87b5d04f9294add7761a2afbfd already in progress: creating tablet
I20251026 02:56:50.207101 2377 log.cc:1199] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/1fb2ff87cf0248a0a4a77f2733ef867b
I20251026 02:56:50.207533 2377 ts_tablet_manager.cc:1950] T 1fb2ff87cf0248a0a4a77f2733ef867b P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.208395 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1068) took 3749 ms. Trace:
I20251026 02:56:50.207007 2381 tablet_bootstrap.cc:492] T cfac07917ee34e6dba3fac5e081f6b43 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:50.208680 2381 ts_tablet_manager.cc:1403] T cfac07917ee34e6dba3fac5e081f6b43 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.008s
I20251026 02:56:50.209456 2381 raft_consensus.cc:359] T cfac07917ee34e6dba3fac5e081f6b43 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.209661 2381 raft_consensus.cc:385] T cfac07917ee34e6dba3fac5e081f6b43 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.209735 2381 raft_consensus.cc:740] T cfac07917ee34e6dba3fac5e081f6b43 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.209949 2381 consensus_queue.cc:260] T cfac07917ee34e6dba3fac5e081f6b43 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:50.213842 2053 consensus_peers.cc:597] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:50.214078 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 81203.9us
W20251026 02:56:50.214304 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet e45600a92f8e46b096badd700779856d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e45600a92f8e46b096badd700779856d already in progress: creating tablet
I20251026 02:56:50.218376 2378 tablet_bootstrap.cc:654] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.212950 2381 ts_tablet_manager.cc:1434] T cfac07917ee34e6dba3fac5e081f6b43 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:50.219389 2372 tablet_replica.cc:333] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:50.219588 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 283939e86520421b839f9c1af0ddeba3 (DEFAULT_TABLE table=default.Table_63a0dc3ab9a14e37b44c2d6c776ca543 [id=ac0c1c86695443c6b35dc5143de6da36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.219587 2372 raft_consensus.cc:2243] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:50.220009 2372 raft_consensus.cc:2272] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.220184 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283939e86520421b839f9c1af0ddeba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.220531 2372 ts_tablet_manager.cc:1916] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.225365 2381 tablet_bootstrap.cc:492] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:50.214501 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet d90e62ae86b149b291dc29cf7442c6bc because the tablet was not found. No further retry: Not found: Tablet not found: d90e62ae86b149b291dc29cf7442c6bc
W20251026 02:56:50.226017 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 5c6e3e47b3e7421e945cfbe671b06abe because the tablet was not found. No further retry: Not found: Tablet not found: 5c6e3e47b3e7421e945cfbe671b06abe
W20251026 02:56:50.226202 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ce4092b7ccc4888b5cbe3b38bee2e36 already in progress: creating tablet
W20251026 02:56:50.226383 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3865399cf6934bc18e16ae2cb4887197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3865399cf6934bc18e16ae2cb4887197 already in progress: creating tablet
W20251026 02:56:50.226569 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 4e1291f47f384ff38eadd16040db65b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e1291f47f384ff38eadd16040db65b8 already in progress: creating tablet
W20251026 02:56:50.226742 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 94d5faa4584a4e36b38bdd552d641994 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94d5faa4584a4e36b38bdd552d641994 already in progress: creating tablet
W20251026 02:56:50.226905 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7801f073d5d94076af9299492f86fad2 because the tablet was not found. No further retry: Not found: Tablet not found: 7801f073d5d94076af9299492f86fad2
I20251026 02:56:50.227042 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 34c6a7c774044411a4733b115080c4c6 (table default.Table_6b9199ff4e884b4891446da6e6e6195d [id=d6f9d1f56c154ea3a78a6677347c4a05]) successfully deleted
I20251026 02:56:50.227130 1975 raft_consensus.cc:3060] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.227187 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet a012d868a78f4324a92db83c81931b9d (table default.Table_3dd70def7f1046ea9f2b5bbb7a74e4cb [id=b4e3ea5575a140ac85b4c0cbd57c9ec4]) successfully deleted
I20251026 02:56:50.227572 2381 tablet_bootstrap.cc:654] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.228415 2377 rpcz_store.cc:276] 1026 02:56:46.459238 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.459313 (+ 75us) service_pool.cc:225] Handling call
1026 02:56:50.188908 (+3729595us) ts_tablet_manager.cc:1151] Deleting tablet 1fb2ff87cf0248a0a4a77f2733ef867b
1026 02:56:50.188920 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.192808 (+ 3888us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.203244 (+ 10436us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.208368 (+ 5124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3729332}
I20251026 02:56:50.228901 2377 tablet_replica.cc:333] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.229075 2377 raft_consensus.cc:2243] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:50.229223 2377 raft_consensus.cc:2272] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.230887 2377 ts_tablet_manager.cc:1916] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.232576 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 06de594e17244f0d906ba296231358d4 (DEFAULT_TABLE table=default.Table_33747351d5ee48bfb112182b05f47bee [id=9a75911c4a1d4e0393e9473028df45e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.232944 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06de594e17244f0d906ba296231358d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.235001 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 2386155307964833b90565b6da44ab8b (DEFAULT_TABLE table=default.Table_4e171e396e77413bb4d15fb2bcf9b0e5 [id=a959b24dbafa425da918d1d26c739ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.235318 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2386155307964833b90565b6da44ab8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.235826 4256 consensus_queue.cc:1048] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:50.239490 2381 tablet_bootstrap.cc:492] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:50.239663 2381 ts_tablet_manager.cc:1403] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20251026 02:56:50.240365 2381 raft_consensus.cc:359] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.240582 2381 raft_consensus.cc:385] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.240661 2381 raft_consensus.cc:740] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.240880 2381 consensus_queue.cc:260] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.241389 2381 ts_tablet_manager.cc:1434] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:50.241691 2381 tablet_bootstrap.cc:492] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:50.242269 2378 tablet_bootstrap.cc:492] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:50.242410 2378 ts_tablet_manager.cc:1403] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20251026 02:56:50.243060 2378 raft_consensus.cc:359] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.243216 2378 raft_consensus.cc:385] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.243269 2378 raft_consensus.cc:740] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.243451 2378 consensus_queue.cc:260] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.243561 2381 tablet_bootstrap.cc:654] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.243945 2378 ts_tablet_manager.cc:1434] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20251026 02:56:50.245158 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 3b7d85a2941545ceaa16b22e8328b4ff because the tablet was not found. No further retry: Not found: Tablet not found: 3b7d85a2941545ceaa16b22e8328b4ff
W20251026 02:56:50.245419 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet e701d4023fee45d18c48997aab3227c2 because the tablet was not found. No further retry: Not found: Tablet not found: e701d4023fee45d18c48997aab3227c2
W20251026 02:56:50.245651 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 885c10070d6942f7abe52c4558d887e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 885c10070d6942f7abe52c4558d887e6 already in progress: creating tablet
W20251026 02:56:50.245788 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 9b976c87b5d04f9294add7761a2afbfd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b976c87b5d04f9294add7761a2afbfd already in progress: creating tablet
W20251026 02:56:50.245913 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 6a3491b303f94fe2a2c838297f73d464 because the tablet was not found. No further retry: Not found: Tablet not found: 6a3491b303f94fe2a2c838297f73d464
I20251026 02:56:50.246124 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 62d30d79a6e4432a8dbe40a104e87fa0 (table default.Table_5f7b25fab37c4983acf922bd78374da6 [id=777528a5bf944ecb96ebdbea9e7ee559]) successfully deleted
W20251026 02:56:50.246268 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 142043b5d558490c9399626c3764ff7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 142043b5d558490c9399626c3764ff7e already in progress: creating tablet
W20251026 02:56:50.246398 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 586bd6e40a7a4785b71b2a3d542e2411 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 586bd6e40a7a4785b71b2a3d542e2411 already in progress: creating tablet
W20251026 02:56:50.246559 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 075f6ca8787943a09443863c544a1284 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 075f6ca8787943a09443863c544a1284 already in progress: creating tablet
I20251026 02:56:50.246995 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 4880eb8bc1214e80a2baeac5fa497727 (DEFAULT_TABLE table=default.Table_e1d771f7ba5b433b91c68d98c6e48cf1 [id=b31c19a4727a4647a8bab645f5c784f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.247392 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4880eb8bc1214e80a2baeac5fa497727. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.248176 2094 tablet_service.cc:1505] Processing CreateTablet for tablet ee23940460a64d7ea1fe97758bb85fcb (DEFAULT_TABLE table=default.Table_164cb1784f4645e499339f7163fdfc2b [id=91b4b1a8162441299c680e7c202a96d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.248425 2377 ts_tablet_manager.cc:1929] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.248535 2377 log.cc:1199] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/0cf76ef8f6af445eb0ce15c8d79136c5
I20251026 02:56:50.248600 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee23940460a64d7ea1fe97758bb85fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.248835 2377 ts_tablet_manager.cc:1950] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.249514 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1069) took 3790 ms. Trace:
I20251026 02:56:50.249599 2377 rpcz_store.cc:276] 1026 02:56:46.459331 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.468323 (+ 8992us) service_pool.cc:225] Handling call
1026 02:56:50.228850 (+3760527us) ts_tablet_manager.cc:1151] Deleting tablet 0cf76ef8f6af445eb0ce15c8d79136c5
1026 02:56:50.228862 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.235311 (+ 6449us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.248326 (+ 13015us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.249496 (+ 1170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3752358}
I20251026 02:56:50.249899 2377 tablet_replica.cc:333] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.250089 2377 raft_consensus.cc:2243] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 2 LEADER]: Raft consensus shutting down.
I20251026 02:56:50.250308 2377 pending_rounds.cc:70] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:50.250355 2377 pending_rounds.cc:77] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7214888591548731392 op_type: NO_OP noop_request { }
I20251026 02:56:50.250442 2377 raft_consensus.cc:2889] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:50.250519 2377 raft_consensus.cc:2272] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.251562 2377 ts_tablet_manager.cc:1916] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.255775 2378 tablet_bootstrap.cc:492] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:50.260433 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 7bef60e3a10843b4be2022ab4b64be68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bef60e3a10843b4be2022ab4b64be68 already in progress: creating tablet
I20251026 02:56:50.260777 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 8c1e63ac8c5645f3ba1578792c0a0dae (table default.Table_528af198a403496a8c1b27fa5821123b [id=e6aed02773494e3c8694deedf311adc6]) successfully deleted
I20251026 02:56:50.260941 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 1fb2ff87cf0248a0a4a77f2733ef867b (table default.Table_26257eca63ce4baaa75beae2e9db1340 [id=fbc97817a9584474abc0b2ab2052abb2]) successfully deleted
I20251026 02:56:50.261109 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 0cf76ef8f6af445eb0ce15c8d79136c5 (table default.Table_72ee520a0e4d45fb920d1e3cc7a426b3 [id=33334e177bb94b3e9de2a7e7ee59979b]) successfully deleted
I20251026 02:56:50.262272 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 8ed7a9f510dd4297b581bd0c4f610b48 (DEFAULT_TABLE table=default.Table_f1f3e2329722434cae69c91816fab475 [id=0cedb08dd0064272a185d62c7641f12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.262528 4258 consensus_queue.cc:1048] T 398c1f420974453b9b6b0fe17795378b P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:50.262775 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed7a9f510dd4297b581bd0c4f610b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.263469 2377 ts_tablet_manager.cc:1929] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251026 02:56:50.268368 2377 log.cc:1199] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/9c7efb70c0804c149f36440a4a9e6ef8
W20251026 02:56:50.268713 2195 consensus_peers.cc:597] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a53f25688e3b45dca68984e2c543d094. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:50.268942 2377 ts_tablet_manager.cc:1950] T 9c7efb70c0804c149f36440a4a9e6ef8 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.270156 2378 tablet_bootstrap.cc:654] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.270730 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1067) took 3811 ms. Trace:
I20251026 02:56:50.271975 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 04fae9383a7f45d299b11af554d09df2 (DEFAULT_TABLE table=default.Table_deeb6a15a5424e609e622a6c71b3553a [id=d0d4f71c191b41f8a4240c2fae4d28fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.272076 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 1e9482cdd69647dab6bcc2e1c75e4650 (DEFAULT_TABLE table=default.Table_1eca935882b04878bb1362a022e82ecf [id=732d51d0d5174293985c760d55f6f46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.272228 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 41cd60fe93d44cc4b07789a6288d86d9 (DEFAULT_TABLE table=default.Table_0b10d09d420e46ac9a13bada7fd2f740 [id=0c49b29c6a764cbe91959c5909e09496]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.272408 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 06de594e17244f0d906ba296231358d4 (DEFAULT_TABLE table=default.Table_33747351d5ee48bfb112182b05f47bee [id=9a75911c4a1d4e0393e9473028df45e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.272408 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 7ab71730fb5e44589046a6ff0cb516b7 (DEFAULT_TABLE table=default.Table_e5f896de4e0d426da281ce2ec2724a04 [id=d801ee46b9f442c0a22fa4b142877a98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.272470 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e9482cdd69647dab6bcc2e1c75e4650. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.272670 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 283939e86520421b839f9c1af0ddeba3 (DEFAULT_TABLE table=default.Table_63a0dc3ab9a14e37b44c2d6c776ca543 [id=ac0c1c86695443c6b35dc5143de6da36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.272751 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06de594e17244f0d906ba296231358d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.272889 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41cd60fe93d44cc4b07789a6288d86d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.271324 2377 rpcz_store.cc:276] 1026 02:56:46.459153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.468400 (+ 9247us) service_pool.cc:225] Handling call
1026 02:56:50.249858 (+3781458us) ts_tablet_manager.cc:1151] Deleting tablet 9c7efb70c0804c149f36440a4a9e6ef8
1026 02:56:50.249867 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.255388 (+ 5521us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.263387 (+ 7999us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.270711 (+ 7324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3773163}
I20251026 02:56:50.273618 2377 tablet_replica.cc:333] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.273905 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 1e9482cdd69647dab6bcc2e1c75e4650 (DEFAULT_TABLE table=default.Table_1eca935882b04878bb1362a022e82ecf [id=732d51d0d5174293985c760d55f6f46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.274196 2377 raft_consensus.cc:2243] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:50.274498 2377 raft_consensus.cc:2272] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.274726 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 41cd60fe93d44cc4b07789a6288d86d9 (DEFAULT_TABLE table=default.Table_0b10d09d420e46ac9a13bada7fd2f740 [id=0c49b29c6a764cbe91959c5909e09496]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.275151 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 283939e86520421b839f9c1af0ddeba3 (DEFAULT_TABLE table=default.Table_63a0dc3ab9a14e37b44c2d6c776ca543 [id=ac0c1c86695443c6b35dc5143de6da36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.274451 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 7ab71730fb5e44589046a6ff0cb516b7 (DEFAULT_TABLE table=default.Table_e5f896de4e0d426da281ce2ec2724a04 [id=d801ee46b9f442c0a22fa4b142877a98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.275697 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab71730fb5e44589046a6ff0cb516b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.275871 2377 ts_tablet_manager.cc:1916] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.276005 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04fae9383a7f45d299b11af554d09df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.274231 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e9482cdd69647dab6bcc2e1c75e4650. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.276325 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283939e86520421b839f9c1af0ddeba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.280939 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41cd60fe93d44cc4b07789a6288d86d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.288429 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283939e86520421b839f9c1af0ddeba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.289866 2377 ts_tablet_manager.cc:1929] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.290902 2377 log.cc:1199] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/41b7b0c36ba748f0a1565892dbe98a78
I20251026 02:56:50.291465 2377 ts_tablet_manager.cc:1950] T 41b7b0c36ba748f0a1565892dbe98a78 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.291492 2376 rpcz_store.cc:276] 1026 02:56:46.584914 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.587039 (+ 2125us) service_pool.cc:225] Handling call
1026 02:56:49.923436 (+3336397us) ts_tablet_manager.cc:1151] Deleting tablet 1406c22ee4ee4f94acecad9546208e79
1026 02:56:49.923448 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:49.923556 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3336089}
I20251026 02:56:50.291841 2376 tablet_replica.cc:333] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:50.292058 2376 raft_consensus.cc:2243] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:50.292253 2376 raft_consensus.cc:2272] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.292335 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab71730fb5e44589046a6ff0cb516b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.292790 2376 ts_tablet_manager.cc:1916] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.293473 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1066) took 3870 ms. Trace:
I20251026 02:56:50.293849 2377 rpcz_store.cc:276] 1026 02:56:46.459009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.480055 (+ 21046us) service_pool.cc:225] Handling call
1026 02:56:50.273573 (+3793518us) ts_tablet_manager.cc:1151] Deleting tablet 41b7b0c36ba748f0a1565892dbe98a78
1026 02:56:50.273583 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.279479 (+ 5896us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.289785 (+ 10306us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.293453 (+ 3668us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3793214}
I20251026 02:56:50.294474 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1076) took 3739 ms. Trace:
I20251026 02:56:50.294870 2377 rpcz_store.cc:276] 1026 02:56:46.554937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.555344 (+ 407us) service_pool.cc:225] Handling call
1026 02:56:50.294360 (+3739016us) ts_tablet_manager.cc:1151] Deleting tablet 51a39824f5404d45b1756a2581aa1da4
1026 02:56:50.294369 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.294459 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3738770}
I20251026 02:56:50.295488 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1077) took 3680 ms. Trace:
I20251026 02:56:50.295872 2377 rpcz_store.cc:276] 1026 02:56:46.615427 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.654690 (+ 39263us) service_pool.cc:225] Handling call
1026 02:56:50.295381 (+3640691us) ts_tablet_manager.cc:1151] Deleting tablet 1406c22ee4ee4f94acecad9546208e79
1026 02:56:50.295390 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.295470 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3640165}
I20251026 02:56:50.296380 2377 tablet_replica.cc:333] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.296733 2377 raft_consensus.cc:2243] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:50.297554 2377 raft_consensus.cc:2272] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.298480 2377 ts_tablet_manager.cc:1916] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.299499 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 5983c01631e9479d9f5e4ba3b7303909 (DEFAULT_TABLE table=default.Table_03d048a39e0548078a18eb96c4830780 [id=758c8ba786a74595b92325d112fde0ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.300431 2081 tablet_service.cc:1505] Processing CreateTablet for tablet b8af5545da3e4241a588465b1783a8eb (DEFAULT_TABLE table=default.Table_3597ea2076244c4ba6e67599f245a639 [id=4ccfea55427149d98986456d6f5ef970]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.303095 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 9c7efb70c0804c149f36440a4a9e6ef8 (table default.Table_e99391c4c0f74eb3ac3b62cedada8612 [id=4397550364664316a6f1e96fb17d9509]) successfully deleted
I20251026 02:56:50.304987 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 4880eb8bc1214e80a2baeac5fa497727 (DEFAULT_TABLE table=default.Table_e1d771f7ba5b433b91c68d98c6e48cf1 [id=b31c19a4727a4647a8bab645f5c784f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.305181 2224 tablet_service.cc:1505] Processing CreateTablet for tablet b8af5545da3e4241a588465b1783a8eb (DEFAULT_TABLE table=default.Table_3597ea2076244c4ba6e67599f245a639 [id=4ccfea55427149d98986456d6f5ef970]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.305385 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4880eb8bc1214e80a2baeac5fa497727. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.306398 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8af5545da3e4241a588465b1783a8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.306576 2237 tablet_service.cc:1505] Processing CreateTablet for tablet ee23940460a64d7ea1fe97758bb85fcb (DEFAULT_TABLE table=default.Table_164cb1784f4645e499339f7163fdfc2b [id=91b4b1a8162441299c680e7c202a96d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.306841 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee23940460a64d7ea1fe97758bb85fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.307262 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 04fae9383a7f45d299b11af554d09df2 (DEFAULT_TABLE table=default.Table_deeb6a15a5424e609e622a6c71b3553a [id=d0d4f71c191b41f8a4240c2fae4d28fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.307329 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 5983c01631e9479d9f5e4ba3b7303909 (DEFAULT_TABLE table=default.Table_03d048a39e0548078a18eb96c4830780 [id=758c8ba786a74595b92325d112fde0ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.307554 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04fae9383a7f45d299b11af554d09df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.307675 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 2386155307964833b90565b6da44ab8b (DEFAULT_TABLE table=default.Table_4e171e396e77413bb4d15fb2bcf9b0e5 [id=a959b24dbafa425da918d1d26c739ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.308451 1946 tablet_service.cc:1505] Processing CreateTablet for tablet b8af5545da3e4241a588465b1783a8eb (DEFAULT_TABLE table=default.Table_3597ea2076244c4ba6e67599f245a639 [id=4ccfea55427149d98986456d6f5ef970]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.309959 1663 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_30611e161c274e8b99bd1251e5e527aa [id=b5d91d3e8a2643c09fee85976972391a] alter complete (version 1)
I20251026 02:56:50.310758 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8af5545da3e4241a588465b1783a8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.312088 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 4880eb8bc1214e80a2baeac5fa497727 (DEFAULT_TABLE table=default.Table_e1d771f7ba5b433b91c68d98c6e48cf1 [id=b31c19a4727a4647a8bab645f5c784f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.312472 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4880eb8bc1214e80a2baeac5fa497727. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.316707 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 2386155307964833b90565b6da44ab8b (DEFAULT_TABLE table=default.Table_4e171e396e77413bb4d15fb2bcf9b0e5 [id=a959b24dbafa425da918d1d26c739ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.317030 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2386155307964833b90565b6da44ab8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.320600 1943 tablet_service.cc:1505] Processing CreateTablet for tablet ee23940460a64d7ea1fe97758bb85fcb (DEFAULT_TABLE table=default.Table_164cb1784f4645e499339f7163fdfc2b [id=91b4b1a8162441299c680e7c202a96d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.322722 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee23940460a64d7ea1fe97758bb85fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.323873 3997 raft_consensus.cc:493] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.324059 3997 raft_consensus.cc:515] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.324925 3997 leader_election.cc:290] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:50.325755 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fe03778a1e842829cab25c90597a3cd" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:50.328590 2053 leader_election.cc:343] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.331018 2377 ts_tablet_manager.cc:1929] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:50.331580 2377 log.cc:1199] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/3bcd7dec9c2149069e48dcf5e564595e
I20251026 02:56:50.332047 2377 ts_tablet_manager.cc:1950] T 3bcd7dec9c2149069e48dcf5e564595e P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.339568 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fe03778a1e842829cab25c90597a3cd" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:50.339881 4305 consensus_queue.cc:1048] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:50.340253 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2386155307964833b90565b6da44ab8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.340441 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5983c01631e9479d9f5e4ba3b7303909. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:50.343748 1662 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1104345f80d6418fa907a8a26924c281 consensus state has no leader
I20251026 02:56:50.344110 1662 master_service.cc:954] ReplaceTablet: received request to replace tablet 740f7c7f186f45ceaa4c40e76419e3a1 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:50.344964 2053 leader_election.cc:343] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.345093 2053 leader_election.cc:304] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:50.345525 3997 raft_consensus.cc:2749] T 8fe03778a1e842829cab25c90597a3cd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:50.346037 1661 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_7cc1906991304bd4980fe66c79599f73 [id=8203d96f9a7a4a6b9a69234b583600c2] alter complete (version 2)
I20251026 02:56:50.347704 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 59f5078f03224ef18ef4b6864be7122b
I20251026 02:56:50.351442 1663 master_service.cc:954] ReplaceTablet: received request to replace tablet 1dbbfb77851d443f8cf4d798f3e07e59 from {username='slave'} at 127.0.0.1:39910
W20251026 02:56:50.351780 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3513c740306a4c4c97a4e33849775fdc consensus state has no leader
W20251026 02:56:50.353049 1915 consensus_peers.cc:597] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Couldn't send request to peer 696476265495492790739ec126a08539. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62d30d79a6e4432a8dbe40a104e87fa0. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:50.354002 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 848e414f6e134761818039f7e98500ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.354250 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 351cd60b076a49949f03640c981845c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.354374 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 59f5078f03224ef18ef4b6864be7122b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.356514 3997 raft_consensus.cc:493] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.356632 3997 raft_consensus.cc:515] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.357158 3997 leader_election.cc:290] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:50.358438 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64415f5588db4df0a94676b650fb3631" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:50.359416 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 5983c01631e9479d9f5e4ba3b7303909 (DEFAULT_TABLE table=default.Table_03d048a39e0548078a18eb96c4830780 [id=758c8ba786a74595b92325d112fde0ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.360208 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5983c01631e9479d9f5e4ba3b7303909. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.360723 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3e40ba9b6c944a3899576ca411fc1ffb" } modify_external_catalogs: true
I20251026 02:56:50.360980 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3e40ba9b6c944a3899576ca411fc1ffb" } modify_external_catalogs: true
W20251026 02:56:50.361341 2053 leader_election.cc:343] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.361960 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64415f5588db4df0a94676b650fb3631" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:50.362886 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d72028b7f087490dbd200e4172900e7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.363428 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d72028b7f087490dbd200e4172900e7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:50.363416 2053 leader_election.cc:343] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.363535 2053 leader_election.cc:304] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:50.364048 3997 raft_consensus.cc:2749] T 64415f5588db4df0a94676b650fb3631 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:50.366089 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5e8fcf7ea82a40c4bf2e9a09300d7b97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.366734 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e8fcf7ea82a40c4bf2e9a09300d7b97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.368309 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d08c9c1fe775483492716153dba0072d" } modify_external_catalogs: true
I20251026 02:56:50.368453 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d08c9c1fe775483492716153dba0072d" } modify_external_catalogs: true
I20251026 02:56:50.368908 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1088) took 3493 ms. Trace:
I20251026 02:56:50.369019 2377 rpcz_store.cc:276] 1026 02:56:46.875055 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.875121 (+ 66us) service_pool.cc:225] Handling call
1026 02:56:50.296345 (+3421224us) ts_tablet_manager.cc:1151] Deleting tablet 3bcd7dec9c2149069e48dcf5e564595e
1026 02:56:50.296352 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.324424 (+ 28072us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.330909 (+ 6485us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.368886 (+ 37977us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3421008}
I20251026 02:56:50.369465 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1094) took 3294 ms. Trace:
I20251026 02:56:50.369544 2377 rpcz_store.cc:276] 1026 02:56:47.074651 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.074711 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:50.369345 (+3294634us) ts_tablet_manager.cc:1151] Deleting tablet f80b5c4c4e2d4c14a8a5292a05127bc1
1026 02:56:50.369354 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.369448 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3294404}
I20251026 02:56:50.369879 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1096) took 3070 ms. Trace:
I20251026 02:56:50.369810 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_01bfed50414a4ccbba2a65baf1a055cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:50.369932 2377 rpcz_store.cc:276] 1026 02:56:47.299128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.312463 (+ 13335us) service_pool.cc:225] Handling call
1026 02:56:50.369784 (+3057321us) ts_tablet_manager.cc:1151] Deleting tablet 68ad3d4a2d484badb4b35884517b72f1
1026 02:56:50.369793 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.369866 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3056980}
I20251026 02:56:50.370139 2377 tablet_replica.cc:333] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539: stopping tablet replica
W20251026 02:56:50.370276 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01bfed50414a4ccbba2a65baf1a055cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.370329 2377 raft_consensus.cc:2243] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:50.370599 2377 pending_rounds.cc:70] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:50.370640 2377 pending_rounds.cc:77] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888584003710976 op_type: NO_OP noop_request { }
I20251026 02:56:50.370724 2377 raft_consensus.cc:2889] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:50.370803 2377 raft_consensus.cc:2272] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.371351 2377 ts_tablet_manager.cc:1916] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.372697 4305 raft_consensus.cc:493] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.372844 4305 raft_consensus.cc:515] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.373461 4305 leader_election.cc:290] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:50.374204 1916 leader_election.cc:343] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: e89a98fcfdbb4011876b0ebc2af91f67
W20251026 02:56:50.374742 2197 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.Ping -> {remote=127.1.150.61:37655, user_credentials={real_user=slave}} blocked reactor thread for 107571us
I20251026 02:56:50.374825 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e89a98fcfdbb4011876b0ebc2af91f67" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:50.374884 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_1c705e5c947c4376b1d4ecdd4a12ac95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:50.374987 2262 raft_consensus.cc:2468] T e89a98fcfdbb4011876b0ebc2af91f67 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
W20251026 02:56:50.375402 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c705e5c947c4376b1d4ecdd4a12ac95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.377043 1915 leader_election.cc:304] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:50.378322 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c4061134bbfd4244a23d6c37781dcb86" } modify_external_catalogs: true
I20251026 02:56:50.378470 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c4061134bbfd4244a23d6c37781dcb86" } modify_external_catalogs: true
I20251026 02:56:50.378903 2377 ts_tablet_manager.cc:1929] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.379071 2377 log.cc:1199] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/d43b6f22dd4f4f9c9657f21514550312
I20251026 02:56:50.379491 2377 ts_tablet_manager.cc:1950] T d43b6f22dd4f4f9c9657f21514550312 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.380607 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1098) took 2992 ms. Trace:
I20251026 02:56:50.380741 2377 rpcz_store.cc:276] 1026 02:56:47.388586 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.391873 (+ 3287us) service_pool.cc:225] Handling call
1026 02:56:50.370103 (+2978230us) ts_tablet_manager.cc:1151] Deleting tablet d43b6f22dd4f4f9c9657f21514550312
1026 02:56:50.370110 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.375159 (+ 5049us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.378797 (+ 3638us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.380587 (+ 1790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2962559}
I20251026 02:56:50.381126 2377 tablet_replica.cc:333] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.381376 2377 raft_consensus.cc:2243] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:50.381800 2377 raft_consensus.cc:2272] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.382938 2377 ts_tablet_manager.cc:1916] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.383116 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 06de594e17244f0d906ba296231358d4 (DEFAULT_TABLE table=default.Table_33747351d5ee48bfb112182b05f47bee [id=9a75911c4a1d4e0393e9473028df45e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.383842 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e4bfd02ebda04e118feee745199d6bbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:50.384255 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06de594e17244f0d906ba296231358d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:50.384413 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4bfd02ebda04e118feee745199d6bbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.386186 4305 raft_consensus.cc:2804] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:50.386292 4305 raft_consensus.cc:493] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:50.386380 4305 raft_consensus.cc:3060] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.389674 4305 raft_consensus.cc:515] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.389855 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5983c01631e9479d9f5e4ba3b7303909. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.392612 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8af5545da3e4241a588465b1783a8eb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:50.395269 1916 leader_election.cc:343] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: e89a98fcfdbb4011876b0ebc2af91f67
I20251026 02:56:50.398185 2377 ts_tablet_manager.cc:1929] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.398937 1663 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 398c1f420974453b9b6b0fe17795378b (table default.Table_a92e68d05f4b455d810180ee4cc5874d [id=7f64fbec3582415a8924d01e14856a56]). Expected version 1 got 0
I20251026 02:56:50.400422 2377 log.cc:1199] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/95c9fbae62db4850b0f02a2443a3432b
I20251026 02:56:50.401090 3997 consensus_queue.cc:1048] T e203fe0403594d30b44d531153d42a10 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:50.401413 2377 ts_tablet_manager.cc:1950] T 95c9fbae62db4850b0f02a2443a3432b P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.401530 4305 leader_election.cc:290] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:50.402271 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1097) took 3038 ms. Trace:
I20251026 02:56:50.402606 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e89a98fcfdbb4011876b0ebc2af91f67" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:50.402766 2262 raft_consensus.cc:3060] T e89a98fcfdbb4011876b0ebc2af91f67 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.402560 2377 rpcz_store.cc:276] 1026 02:56:47.363405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.382047 (+ 18642us) service_pool.cc:225] Handling call
1026 02:56:50.381081 (+2999034us) ts_tablet_manager.cc:1151] Deleting tablet 95c9fbae62db4850b0f02a2443a3432b
1026 02:56:50.381091 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.391640 (+ 10549us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.398057 (+ 6417us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.402254 (+ 4197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2973495}
I20251026 02:56:50.403409 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1099) took 3007 ms. Trace:
I20251026 02:56:50.403788 2377 rpcz_store.cc:276] 1026 02:56:47.396111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.396170 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:50.403295 (+3007125us) ts_tablet_manager.cc:1151] Deleting tablet 0aabfe11dedd454f8221fb5de25a268b
1026 02:56:50.403306 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.403394 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2995677}
I20251026 02:56:50.404628 2377 tablet_replica.cc:333] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.405455 2262 raft_consensus.cc:2468] T e89a98fcfdbb4011876b0ebc2af91f67 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:50.405620 2378 tablet_bootstrap.cc:492] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:50.405755 2378 ts_tablet_manager.cc:1403] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.162s user 0.004s sys 0.000s
I20251026 02:56:50.406328 2378 raft_consensus.cc:359] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.406688 2378 raft_consensus.cc:385] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.406744 2378 raft_consensus.cc:740] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.406925 2378 consensus_queue.cc:260] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.407374 2378 ts_tablet_manager.cc:1434] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251026 02:56:50.407362 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 1406c22ee4ee4f94acecad9546208e79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1406c22ee4ee4f94acecad9546208e79 already in progress: creating tablet
I20251026 02:56:50.407626 2378 tablet_bootstrap.cc:492] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:50.408277 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 41b7b0c36ba748f0a1565892dbe98a78 (table default.Table_292ad60362294da0976fd8fe6b180c0a [id=4651ecfc8aed4daf99de5a271e8afcda]) successfully deleted
I20251026 02:56:50.409065 2378 tablet_bootstrap.cc:654] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:50.409317 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 51a39824f5404d45b1756a2581aa1da4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51a39824f5404d45b1756a2581aa1da4 already in progress: creating tablet
I20251026 02:56:50.409886 1915 leader_election.cc:304] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:50.410306 4305 raft_consensus.cc:2804] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:50.410435 4305 raft_consensus.cc:697] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:50.410898 4305 consensus_queue.cc:237] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.412761 2377 raft_consensus.cc:2243] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
W20251026 02:56:50.414086 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 1406c22ee4ee4f94acecad9546208e79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1406c22ee4ee4f94acecad9546208e79 already in progress: creating tablet
I20251026 02:56:50.420982 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 3bcd7dec9c2149069e48dcf5e564595e (table default.Table_7eb7fd8131cc4bce81b6cc09c5062fc0 [id=5cbfb8198a8947b3abd1c0801e92f100]) successfully deleted
W20251026 02:56:50.422431 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet f80b5c4c4e2d4c14a8a5292a05127bc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f80b5c4c4e2d4c14a8a5292a05127bc1 already in progress: creating tablet
W20251026 02:56:50.423218 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 68ad3d4a2d484badb4b35884517b72f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68ad3d4a2d484badb4b35884517b72f1 already in progress: creating tablet
I20251026 02:56:50.423609 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet d43b6f22dd4f4f9c9657f21514550312 (table default.Table_528abc1c5aa443a89795cc31b809be9a [id=c5c5c6a68e6848669ecbf8e3dc5fc6fa]) successfully deleted
I20251026 02:56:50.423965 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 95c9fbae62db4850b0f02a2443a3432b (table default.Table_65854c69bc7b4af88efa140b222d480b [id=472d44e7e6f0409ab9d238c4913cc8a8]) successfully deleted
W20251026 02:56:50.424407 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 0aabfe11dedd454f8221fb5de25a268b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0aabfe11dedd454f8221fb5de25a268b already in progress: creating tablet
I20251026 02:56:50.425532 2376 ts_tablet_manager.cc:1929] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.429569 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 59f5078f03224ef18ef4b6864be7122b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.430109 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 62d30d79a6e4432a8dbe40a104e87fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.430327 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet bc4ca4423e3a4e0c96846babec931fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.430496 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cf76ef8f6af445eb0ce15c8d79136c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.430635 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet e89a98fcfdbb4011876b0ebc2af91f67 with delete_type TABLET_DATA_DELETED (replaced by tablet d0f1d6967931482e9de40b0844da6315) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.430759 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b77bb6b4722449c9699b548a9ef6314 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.431247 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 87895fb88c284a4f8d7c58a5ee48ba8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.442435 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 77d4998abe3a4a53a980b6d94b0021a9 (DEFAULT_TABLE table=default.Table_c4c451a485f24525b3dbb3f3626a01e7 [id=5225751c079f4b7cbd21bc817c91851e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.443158 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d4998abe3a4a53a980b6d94b0021a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.443630 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 46f28f2f55ac41249599e4b9a4a53890 (DEFAULT_TABLE table=default.Table_fda5e7b00a8a4e9ba95382d569d8c421 [id=b10cfa3fcea04a5cb33be14a094a1482]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.443980 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f28f2f55ac41249599e4b9a4a53890. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.447762 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 9250d3f2cc354d86881fbc2b082c508b (DEFAULT_TABLE table=default.Table_843ed42261d54cfb917339822d4b30dd [id=f70d32cc88ba40eabbf3747df8b0e6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.448097 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9250d3f2cc354d86881fbc2b082c508b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.448319 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 46f28f2f55ac41249599e4b9a4a53890 (DEFAULT_TABLE table=default.Table_fda5e7b00a8a4e9ba95382d569d8c421 [id=b10cfa3fcea04a5cb33be14a094a1482]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.448694 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f28f2f55ac41249599e4b9a4a53890. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.453263 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet cef68c4d9a3e4a85acdab446bd6082ce
I20251026 02:56:50.454305 4268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "209968db4dee47c49497a66c3ee750b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } } }
I20251026 02:56:50.454464 4268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 [sys.catalog]: This master's current role is: LEADER
I20251026 02:56:50.456432 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 0b04c71ba1cd429d90ff8e451ed45870
I20251026 02:56:50.456724 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet cef68c4d9a3e4a85acdab446bd6082ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.456913 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_860f88c3c76242d28c07e73bcbb7d077" } new_table_name: "default.Table_d4e275a10a7a465dade3d9aae88506d5" modify_external_catalogs: true
I20251026 02:56:50.457931 1656 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 398c1f420974453b9b6b0fe17795378b (table default.Table_a92e68d05f4b455d810180ee4cc5874d [id=7f64fbec3582415a8924d01e14856a56]). Expected version 1 got 0
I20251026 02:56:50.458150 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 740f7c7f186f45ceaa4c40e76419e3a1
I20251026 02:56:50.458258 1656 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet cfac07917ee34e6dba3fac5e081f6b43 (table default.Table_907991c2b0034c118065ead079f273f9 [id=a8d490f3dbc948d38c54d14e178b736e]). Expected version 1 got 0
I20251026 02:56:50.458624 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3ea9b77cd2cc43b99bdf4ab495a678d9" } modify_external_catalogs: true
I20251026 02:56:50.458750 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3ea9b77cd2cc43b99bdf4ab495a678d9" } modify_external_catalogs: true
I20251026 02:56:50.459422 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 9250d3f2cc354d86881fbc2b082c508b (DEFAULT_TABLE table=default.Table_843ed42261d54cfb917339822d4b30dd [id=f70d32cc88ba40eabbf3747df8b0e6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.459822 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9250d3f2cc354d86881fbc2b082c508b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.459954 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet f7da859119bd42ea9fa1290b3d4bf88c
I20251026 02:56:50.460510 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_a86131ef9a344c92952a42fed41f843e" } new_table_name: "default.Table_38b5dde8e9444e1baa99c522d4947641" modify_external_catalogs: true
I20251026 02:56:50.461892 1663 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_6a6edd8b559245748a47a6539dbd2949 [id=9e897d33e6374ad4bc3c8acaae512645] alter complete (version 3)
I20251026 02:56:50.462917 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1fa437ef7f89450bb6f672a1fb491baa" } new_table_name: "default.Table_08cc2df4f16e47898c1191ec1838143c" modify_external_catalogs: true
I20251026 02:56:50.464890 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 77d4998abe3a4a53a980b6d94b0021a9 (DEFAULT_TABLE table=default.Table_c4c451a485f24525b3dbb3f3626a01e7 [id=5225751c079f4b7cbd21bc817c91851e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.465350 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d4998abe3a4a53a980b6d94b0021a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.469182 2375 tablet_bootstrap.cc:492] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:50.469331 2375 ts_tablet_manager.cc:1403] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.318s user 0.004s sys 0.000s
I20251026 02:56:50.469481 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fe03778a1e842829cab25c90597a3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.470039 2375 raft_consensus.cc:359] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.470185 2375 raft_consensus.cc:385] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.470343 2375 raft_consensus.cc:740] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.470557 2375 consensus_queue.cc:260] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.471328 2375 ts_tablet_manager.cc:1434] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:50.471580 2377 raft_consensus.cc:2272] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.471966 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_7a245074eac243e29b301b5f1c84a32d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:50.472128 2377 ts_tablet_manager.cc:1916] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.472329 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet a4f4d7f87916425999574fd5b9f2fdf7 with delete_type TABLET_DATA_DELETED (replaced by tablet ab923e219ff442939b66f1ad0902151c) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.472514 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet 64415f5588db4df0a94676b650fb3631 with delete_type TABLET_DATA_DELETED (replaced by tablet d0b0954ff4824ee7b47dce18bdc61122) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.474076 2229 tablet_service.cc:1505] Processing CreateTablet for tablet b0b27d526b83492cbfa22604c943f76c (DEFAULT_TABLE table=default.Table_85a72fb0164a4922a649a43010b4998a [id=f6ff8dff403644e2bfe2800318a3c1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.474452 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0b27d526b83492cbfa22604c943f76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.475283 1659 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 398c1f420974453b9b6b0fe17795378b (table default.Table_a92e68d05f4b455d810180ee4cc5874d [id=7f64fbec3582415a8924d01e14856a56]). Expected version 1 got 0
I20251026 02:56:50.475836 1661 master_service.cc:954] ReplaceTablet: received request to replace tablet d48c9c2f2b604f39bc6f21bc054c0fad from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:50.476440 1662 catalog_manager.cc:6719] ReplaceTablet: tablet 740f7c7f186f45ceaa4c40e76419e3a1 deleted and replaced by tablet 232fd4092e9a4a7a9fc9200d29180157
I20251026 02:56:50.477766 1659 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 41a1442fffe54add93666e3c78a0cc58 (table default.Table_1faa0c8151724c5e816546ee2e1d5182 [id=a106f9ece0b9467a8bc1d5d4b3524c41]). Expected version 1 got 0
W20251026 02:56:50.478160 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a245074eac243e29b301b5f1c84a32d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.478947 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 740f7c7f186f45ceaa4c40e76419e3a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 232fd4092e9a4a7a9fc9200d29180157) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.479149 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b04c71ba1cd429d90ff8e451ed45870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.479275 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet f7da859119bd42ea9fa1290b3d4bf88c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.479657 4098 raft_consensus.cc:493] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.479773 4098 raft_consensus.cc:515] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.480309 4098 leader_election.cc:290] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:50.486445 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4f4d7f87916425999574fd5b9f2fdf7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:50.487003 2377 ts_tablet_manager.cc:1929] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.487142 2377 log.cc:1199] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/99d318aa0d1847c58167bf0af9e75aa2
W20251026 02:56:50.487144 2053 leader_election.cc:343] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.487376 2081 tablet_service.cc:1505] Processing CreateTablet for tablet b0b27d526b83492cbfa22604c943f76c (DEFAULT_TABLE table=default.Table_85a72fb0164a4922a649a43010b4998a [id=f6ff8dff403644e2bfe2800318a3c1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.487493 2377 ts_tablet_manager.cc:1950] T 99d318aa0d1847c58167bf0af9e75aa2 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.489526 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1100) took 3047 ms. Trace:
I20251026 02:56:50.489622 2377 rpcz_store.cc:276] 1026 02:56:47.442025 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.442089 (+ 64us) service_pool.cc:225] Handling call
1026 02:56:50.404415 (+2962326us) ts_tablet_manager.cc:1151] Deleting tablet 99d318aa0d1847c58167bf0af9e75aa2
1026 02:56:50.404422 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.477362 (+ 72940us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.486897 (+ 9535us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.489507 (+ 2610us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1,"tablet-delete.queue_time_us":2962104}
I20251026 02:56:50.491044 2377 tablet_replica.cc:333] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.491236 2377 raft_consensus.cc:2243] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:50.491535 2377 raft_consensus.cc:2272] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.492154 2377 ts_tablet_manager.cc:1916] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.495419 4255 raft_consensus.cc:493] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.495592 4255 raft_consensus.cc:515] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.496165 4255 leader_election.cc:290] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:50.496693 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc4ca4423e3a4e0c96846babec931fec" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:50.496881 2109 raft_consensus.cc:2468] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:50.497385 2195 leader_election.cc:304] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:50.500404 4255 raft_consensus.cc:2804] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:50.500535 4255 raft_consensus.cc:493] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:50.500648 4255 raft_consensus.cc:3060] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.501963 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0b27d526b83492cbfa22604c943f76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.503208 4255 raft_consensus.cc:515] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.503744 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet c1dd78f088f84a509c761c934c039016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.504611 4255 leader_election.cc:290] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:50.505131 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc4ca4423e3a4e0c96846babec931fec" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:50.505316 2109 raft_consensus.cc:3060] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.507975 2109 raft_consensus.cc:2468] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:50.508795 2195 leader_election.cc:304] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:50.509073 2377 ts_tablet_manager.cc:1929] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.509102 4067 raft_consensus.cc:2804] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:50.509171 2377 log.cc:1199] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/b2bfd9b55c484ef8a9c6345d56f794fb
I20251026 02:56:50.509196 4067 raft_consensus.cc:697] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:50.509414 4067 consensus_queue.cc:237] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:50.509261 1647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.60:33729, user_credentials={real_user=slave}} blocked reactor thread for 59382.4us
I20251026 02:56:50.512527 2375 tablet_bootstrap.cc:492] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:50.509490 2377 ts_tablet_manager.cc:1950] T b2bfd9b55c484ef8a9c6345d56f794fb P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.513917 2375 tablet_bootstrap.cc:654] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.514487 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1101) took 3056 ms. Trace:
I20251026 02:56:50.514617 2377 rpcz_store.cc:276] 1026 02:56:47.458169 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.458424 (+ 255us) service_pool.cc:225] Handling call
1026 02:56:50.490997 (+3032573us) ts_tablet_manager.cc:1151] Deleting tablet b2bfd9b55c484ef8a9c6345d56f794fb
1026 02:56:50.491006 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.504953 (+ 13947us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.508985 (+ 4032us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.514464 (+ 5479us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3032387}
I20251026 02:56:50.516427 2377 tablet_replica.cc:333] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.517578 2377 raft_consensus.cc:2243] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:50.517760 2377 raft_consensus.cc:2272] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.519047 2377 ts_tablet_manager.cc:1916] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.522161 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet f7da859119bd42ea9fa1290b3d4bf88c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.526028 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet cef68c4d9a3e4a85acdab446bd6082ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.526208 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b04c71ba1cd429d90ff8e451ed45870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.527894 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 740f7c7f186f45ceaa4c40e76419e3a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 232fd4092e9a4a7a9fc9200d29180157) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.425658 1945 tablet_service.cc:1552] Processing DeleteTablet for tablet e89a98fcfdbb4011876b0ebc2af91f67 with delete_type TABLET_DATA_DELETED (replaced by tablet d0f1d6967931482e9de40b0844da6315) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.425662 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dbbfb77851d443f8cf4d798f3e07e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.428807 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 59f5078f03224ef18ef4b6864be7122b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.437391 1962 tablet_service.cc:1505] Processing CreateTablet for tablet b0b27d526b83492cbfa22604c943f76c (DEFAULT_TABLE table=default.Table_85a72fb0164a4922a649a43010b4998a [id=f6ff8dff403644e2bfe2800318a3c1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.438625 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 9250d3f2cc354d86881fbc2b082c508b (DEFAULT_TABLE table=default.Table_843ed42261d54cfb917339822d4b30dd [id=f70d32cc88ba40eabbf3747df8b0e6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.440145 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 77d4998abe3a4a53a980b6d94b0021a9 (DEFAULT_TABLE table=default.Table_c4c451a485f24525b3dbb3f3626a01e7 [id=5225751c079f4b7cbd21bc817c91851e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.481698 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4f4d7f87916425999574fd5b9f2fdf7" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:50.496776 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc4ca4423e3a4e0c96846babec931fec" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:50.501641 4305 consensus_queue.cc:1048] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:50.504587 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc4ca4423e3a4e0c96846babec931fec" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:50.529217 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet a53f25688e3b45dca68984e2c543d094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:50.529359 2194 leader_election.cc:343] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.510205 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet cef68c4d9a3e4a85acdab446bd6082ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.529071 2376 log.cc:1199] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/34c6a7c774044411a4733b115080c4c6
W20251026 02:56:50.529665 2194 leader_election.cc:343] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.529882 1948 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fb42b54f558499090a1d7cf257dd84c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.524072 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b04c71ba1cd429d90ff8e451ed45870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.527163 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 740f7c7f186f45ceaa4c40e76419e3a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 232fd4092e9a4a7a9fc9200d29180157) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.516731 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet f7da859119bd42ea9fa1290b3d4bf88c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.529821 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet 1eabfc95caa84afc9a8956a7ac13fea6 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b8e1d79d2a247dba25540daf6b81bf0) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.517149 2378 tablet_bootstrap.cc:492] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:50.530234 2378 ts_tablet_manager.cc:1403] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.005s
I20251026 02:56:50.530786 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ea364c4a5cf4f3f99d9ff6904add697 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:35450
W20251026 02:56:50.530993 2053 leader_election.cc:343] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.530987 2378 raft_consensus.cc:359] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.531154 2378 raft_consensus.cc:385] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.531208 2378 raft_consensus.cc:740] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.531132 2053 leader_election.cc:304] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:50.531442 4098 raft_consensus.cc:2749] T a4f4d7f87916425999574fd5b9f2fdf7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:50.531399 2378 consensus_queue.cc:260] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.531857 2378 ts_tablet_manager.cc:1434] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:50.532542 2378 tablet_bootstrap.cc:492] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:50.532512 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 99d318aa0d1847c58167bf0af9e75aa2 (table default.Table_62fd5a902d0648cea484f259d02de5f9 [id=3dc4d9941b7147adb84d1754e2783aa9]) successfully deleted
I20251026 02:56:50.532569 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b9a5daa36b547a7888d38dc3329eaf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d06944f71fd4e42a6ea46969c86e9f1) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.532938 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet b2bfd9b55c484ef8a9c6345d56f794fb (table default.Table_30e98b21e65d443097e2fc01f96f4780 [id=0d9be16d1aec415a8f9a2433f8d7ac6b]) successfully deleted
W20251026 02:56:50.539758 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2337224b44bf4bcc9062d4f8940269ab consensus state has no leader
I20251026 02:56:50.540187 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet d48c9c2f2b604f39bc6f21bc054c0fad
W20251026 02:56:50.540321 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36e70ed89e8c422d8448ad3e0dd8b832 consensus state has no leader
I20251026 02:56:50.540307 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet 95c9fbae62db4850b0f02a2443a3432b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.541602 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet aa1253c60ffb4b37849c9318307b94f1
I20251026 02:56:50.541631 1661 catalog_manager.cc:6719] ReplaceTablet: tablet d48c9c2f2b604f39bc6f21bc054c0fad deleted and replaced by tablet 7eb85291b9fd4fed92139084df5eed60
I20251026 02:56:50.542371 2100 tablet_service.cc:1552] Processing DeleteTablet for tablet aa1253c60ffb4b37849c9318307b94f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.542872 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet d48c9c2f2b604f39bc6f21bc054c0fad with delete_type TABLET_DATA_DELETED (replaced by tablet 7eb85291b9fd4fed92139084df5eed60) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.538578 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 46f28f2f55ac41249599e4b9a4a53890 (DEFAULT_TABLE table=default.Table_fda5e7b00a8a4e9ba95382d569d8c421 [id=b10cfa3fcea04a5cb33be14a094a1482]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.544878 1951 tablet_service.cc:1552] Processing DeleteTablet for tablet d48c9c2f2b604f39bc6f21bc054c0fad with delete_type TABLET_DATA_DELETED (replaced by tablet 7eb85291b9fd4fed92139084df5eed60) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.545145 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet aa1253c60ffb4b37849c9318307b94f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.544624 2376 ts_tablet_manager.cc:1950] T 34c6a7c774044411a4733b115080c4c6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:50.547247 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b77bb6b4722449c9699b548a9ef6314 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.548202 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ca4a98dc25144099a2ca90bb12766232"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:50.548146 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_91d47d9643cc4d25a7fb31d6360abd20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.548764 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca4a98dc25144099a2ca90bb12766232 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:50.548974 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91d47d9643cc4d25a7fb31d6360abd20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.550124 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_be9843c93f174ca385806dea17d6d181"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.550619 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be9843c93f174ca385806dea17d6d181 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.559356 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet d48c9c2f2b604f39bc6f21bc054c0fad with delete_type TABLET_DATA_DELETED (replaced by tablet 7eb85291b9fd4fed92139084df5eed60) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.560525 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet aa1253c60ffb4b37849c9318307b94f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.562860 4069 consensus_queue.cc:1048] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:50.562933 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1011) took 3963 ms. Trace:
I20251026 02:56:50.561213 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fb42b54f558499090a1d7cf257dd84c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.578681 2377 ts_tablet_manager.cc:1929] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.578848 2377 log.cc:1199] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/9a345fa5edbd48d2891b8b40b09cb3f8
I20251026 02:56:50.579186 2377 ts_tablet_manager.cc:1950] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.579941 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1102) took 3110 ms. Trace:
I20251026 02:56:50.580031 2377 rpcz_store.cc:276] 1026 02:56:47.469719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.469788 (+ 69us) service_pool.cc:225] Handling call
1026 02:56:50.516375 (+3046587us) ts_tablet_manager.cc:1151] Deleting tablet 9a345fa5edbd48d2891b8b40b09cb3f8
1026 02:56:50.516385 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.547429 (+ 31044us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.578535 (+ 31106us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.579925 (+ 1390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3046377}
I20251026 02:56:50.567880 2376 rpcz_store.cc:276] 1026 02:56:46.599217 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.599298 (+ 81us) service_pool.cc:225] Handling call
1026 02:56:50.291792 (+3692494us) ts_tablet_manager.cc:1151] Deleting tablet 34c6a7c774044411a4733b115080c4c6
1026 02:56:50.291803 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.295558 (+ 3755us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.425407 (+129849us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.562171 (+136764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3692215}
I20251026 02:56:50.580399 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1103) took 3100 ms. Trace:
I20251026 02:56:50.580461 2377 rpcz_store.cc:276] 1026 02:56:47.480467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.480564 (+ 97us) service_pool.cc:225] Handling call
1026 02:56:50.580291 (+3099727us) ts_tablet_manager.cc:1151] Deleting tablet 97aa7f0266854c0a8f17e6bceac0c9b5
1026 02:56:50.580301 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.580386 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3099559}
I20251026 02:56:50.580766 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1104) took 3084 ms. Trace:
I20251026 02:56:50.580798 2376 tablet_replica.cc:333] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:50.580839 2377 rpcz_store.cc:276] 1026 02:56:47.496034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.496435 (+ 401us) service_pool.cc:225] Handling call
1026 02:56:50.580671 (+3084236us) ts_tablet_manager.cc:1151] Deleting tablet f840050f7e314d74ac6e604354a7d4c9
1026 02:56:50.580682 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.580752 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3084048}
I20251026 02:56:50.581053 2377 tablet_replica.cc:333] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.581210 2377 raft_consensus.cc:2243] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:50.581316 4305 raft_consensus.cc:493] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.581450 4305 raft_consensus.cc:515] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.581552 2377 raft_consensus.cc:2272] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.582058 2377 ts_tablet_manager.cc:1916] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.582058 4305 leader_election.cc:290] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:50.582619 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e203fe0403594d30b44d531153d42a10" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:50.583564 2376 raft_consensus.cc:2243] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:50.583662 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e203fe0403594d30b44d531153d42a10" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:50.583628 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 34c6a7c774044411a4733b115080c4c6 (table default.Table_6b9199ff4e884b4891446da6e6e6195d [id=d6f9d1f56c154ea3a78a6677347c4a05]) successfully deleted
I20251026 02:56:50.583895 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 9a345fa5edbd48d2891b8b40b09cb3f8 (table default.Table_3d25c519f16647ca962f3862c6cd4f98 [id=eca037eee4b54221b7c9771c390055c8]) successfully deleted
W20251026 02:56:50.584067 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 97aa7f0266854c0a8f17e6bceac0c9b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97aa7f0266854c0a8f17e6bceac0c9b5 already in progress: creating tablet
I20251026 02:56:50.584180 1915 leader_election.cc:304] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
W20251026 02:56:50.584260 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet f840050f7e314d74ac6e604354a7d4c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f840050f7e314d74ac6e604354a7d4c9 already in progress: creating tablet
I20251026 02:56:50.584512 4305 raft_consensus.cc:2749] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:50.584782 2376 raft_consensus.cc:2272] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.593875 2376 ts_tablet_manager.cc:1916] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.603499 4069 raft_consensus.cc:493] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.603654 4069 raft_consensus.cc:515] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.604249 4069 leader_election.cc:290] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:50.604951 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ea364c4a5cf4f3f99d9ff6904add697" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:50.605172 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ea364c4a5cf4f3f99d9ff6904add697" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:50.604848 4350 tablet.cc:1722] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251026 02:56:50.605538 2195 leader_election.cc:343] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: Tablet not RUNNING: STOPPED
W20251026 02:56:50.605851 2194 leader_election.cc:343] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.606009 2194 leader_election.cc:304] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:50.606243 4258 raft_consensus.cc:2749] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:50.615160 4098 raft_consensus.cc:493] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.615334 4098 raft_consensus.cc:515] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.615919 4098 leader_election.cc:290] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:50.616622 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "848e414f6e134761818039f7e98500ac" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:50.617225 2053 leader_election.cc:343] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.617595 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "848e414f6e134761818039f7e98500ac" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:50.617904 1915 consensus_peers.cc:597] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Couldn't send request to peer 696476265495492790739ec126a08539. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:50.618155 2053 leader_election.cc:343] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.618234 2053 leader_election.cc:304] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:50.618932 4098 raft_consensus.cc:2749] T 848e414f6e134761818039f7e98500ac P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:50.623651 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0b27d526b83492cbfa22604c943f76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.623982 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d4998abe3a4a53a980b6d94b0021a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.624316 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9250d3f2cc354d86881fbc2b082c508b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.627350 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7807f6bf2766414497c3dce95b5319cd" } new_table_name: "default.Table_79d8aa03b0ed42feab9dd85971ae173b" modify_external_catalogs: true
I20251026 02:56:50.630342 2378 tablet_bootstrap.cc:654] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.630282 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5fb9a6abaf3b49ba80d560ab9ca37142"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.630793 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fb9a6abaf3b49ba80d560ab9ca37142 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.631865 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ea91bad0b400465a98d6aec2534bcabe" } modify_external_catalogs: true
I20251026 02:56:50.632014 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ea91bad0b400465a98d6aec2534bcabe" } modify_external_catalogs: true
I20251026 02:56:50.633193 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d4e275a10a7a465dade3d9aae88506d5" } new_table_name: "default.Table_c2f8423ec87244bca7c3bd7f1f971a88" modify_external_catalogs: true
I20251026 02:56:50.632700 2377 ts_tablet_manager.cc:1929] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.633867 2377 log.cc:1199] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/4a3f3fc3c7a04d05bae28472251f3acc
W20251026 02:56:50.633981 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a7b81b4ed2c49bfa1674d5775110545 has no consensus state
I20251026 02:56:50.634200 2377 ts_tablet_manager.cc:1950] T 4a3f3fc3c7a04d05bae28472251f3acc P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.634327 4297 raft_consensus.cc:493] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.634446 4297 raft_consensus.cc:515] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.634956 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1105) took 3117 ms. Trace:
I20251026 02:56:50.635035 4297 leader_election.cc:290] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:50.635028 2377 rpcz_store.cc:276] 1026 02:56:47.517365 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.517747 (+ 382us) service_pool.cc:225] Handling call
1026 02:56:50.581017 (+3063270us) ts_tablet_manager.cc:1151] Deleting tablet 4a3f3fc3c7a04d05bae28472251f3acc
1026 02:56:50.581025 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.599085 (+ 18060us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.632584 (+ 33499us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.634923 (+ 2339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3063106}
I20251026 02:56:50.635393 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 4a3f3fc3c7a04d05bae28472251f3acc (table default.Table_7713a2385fb24bb7bfecfaa78176c128 [id=fae1b7f4879e4d338a4113383721fcc5]) successfully deleted
I20251026 02:56:50.636328 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a1442fffe54add93666e3c78a0cc58" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:50.637137 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a1442fffe54add93666e3c78a0cc58" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:50.638393 1916 leader_election.cc:304] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:50.638644 4297 raft_consensus.cc:2749] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:50.621383 4351 tablet.cc:1722] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251026 02:56:50.639586 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36e70ed89e8c422d8448ad3e0dd8b832 consensus state has no leader
I20251026 02:56:50.639698 4352 tablet.cc:1722] T 5e9b0e1d3f3f4fdbb8c58f26a4308771 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251026 02:56:50.639870 2194 consensus_peers.cc:597] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:50.641161 4258 consensus_queue.cc:1048] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:50.644500 2381 tablet_bootstrap.cc:492] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:50.644686 2381 ts_tablet_manager.cc:1403] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.403s user 0.051s sys 0.092s
I20251026 02:56:50.645421 2381 raft_consensus.cc:359] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.645694 2381 raft_consensus.cc:385] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.645794 2381 raft_consensus.cc:740] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.646037 2381 consensus_queue.cc:260] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.646572 2381 ts_tablet_manager.cc:1434] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:50.648371 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f28f2f55ac41249599e4b9a4a53890. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.650059 4297 consensus_queue.cc:1048] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:50.640324 2377 tablet_replica.cc:333] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:50.650754 2377 raft_consensus.cc:2243] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:50.651055 2377 raft_consensus.cc:2272] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.651623 2377 ts_tablet_manager.cc:1916] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.654134 2381 tablet_bootstrap.cc:492] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:50.655556 2381 tablet_bootstrap.cc:654] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.659060 2375 tablet_bootstrap.cc:492] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:50.659193 2375 ts_tablet_manager.cc:1403] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.147s user 0.002s sys 0.002s
I20251026 02:56:50.660328 2375 raft_consensus.cc:359] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.660503 2375 raft_consensus.cc:385] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.660552 2375 raft_consensus.cc:740] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.668889 2372 ts_tablet_manager.cc:1929] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:50.670794 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 827afd5a50c24795b6e9ccec14046e77
I20251026 02:56:50.672487 1954 tablet_service.cc:1552] Processing DeleteTablet for tablet 827afd5a50c24795b6e9ccec14046e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.674232 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 827afd5a50c24795b6e9ccec14046e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.675175 1657 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet 2fd8f02126b24285bbfa575540ba0f09 (table default.Table_7539d36a088f4eafaa21d151ed9ea1cb [id=1950d7cb826b4a99b1ba8866e1d845f3]). Expected version 1 got 0
I20251026 02:56:50.675427 1657 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet cfac07917ee34e6dba3fac5e081f6b43 (table default.Table_907991c2b0034c118065ead079f273f9 [id=a8d490f3dbc948d38c54d14e178b736e]). Expected version 1 got 0
I20251026 02:56:50.685568 1657 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_08cc2df4f16e47898c1191ec1838143c [id=44c37945ff9f48e8b2c07648394ac177] alter complete (version 2)
W20251026 02:56:50.687364 1647 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet fd9eef082c084267b10163fee196ba6f (table default.Table_04a58bdc201c45eaafcdbcfc8ccf93bd [id=b926a666f79b4eddbc59ec883e675bc2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fd9eef082c084267b10163fee196ba6f consensus state has no leader
I20251026 02:56:50.679809 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 827afd5a50c24795b6e9ccec14046e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.694940 2372 log.cc:1199] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/6ea364c4a5cf4f3f99d9ff6904add697
I20251026 02:56:50.695303 2372 ts_tablet_manager.cc:1950] T 6ea364c4a5cf4f3f99d9ff6904add697 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:50.696017 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1045) took 4640 ms. Trace:
I20251026 02:56:50.696100 2372 rpcz_store.cc:276] 1026 02:56:46.055349 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.055410 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:50.203998 (+4148588us) ts_tablet_manager.cc:1151] Deleting tablet 6ea364c4a5cf4f3f99d9ff6904add697
1026 02:56:50.219344 (+ 15346us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.648396 (+429052us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.668752 (+ 20356us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.695997 (+ 27245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4148368}
I20251026 02:56:50.660743 2375 consensus_queue.cc:260] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.696612 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 6ea364c4a5cf4f3f99d9ff6904add697 (table default.Table_e75c4f87d9d54b479fba42c803c3bc9b [id=02c334c1e9e5480d8ac279c0b5e62800]) successfully deleted
I20251026 02:56:50.696929 2375 ts_tablet_manager.cc:1434] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20251026 02:56:50.697213 2375 tablet_bootstrap.cc:492] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:50.698912 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 76a093eaef6a48ba980ee1f745e4f25f (DEFAULT_TABLE table=default.Table_73c105436cb84727b8cabb2354aae56e [id=eebbff38179b4d8fb5eed1654e6789ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.699040 2375 tablet_bootstrap.cc:654] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.699344 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a093eaef6a48ba980ee1f745e4f25f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:50.710281 1916 consensus_peers.cc:597] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2d97084b5914350afbfda5c2a34bf9a. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:50.713416 1661 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3478623ef36e4ef7910a71260ba0ecb4" } modify_external_catalogs: true
I20251026 02:56:50.713678 1661 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3478623ef36e4ef7910a71260ba0ecb4" } modify_external_catalogs: true
I20251026 02:56:50.718248 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_a3b2085212a547c1ac7c6ad183faa9ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.718796 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3b2085212a547c1ac7c6ad183faa9ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.719076 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6cb0771a15564fff8f171eaadc706272"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.719514 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cb0771a15564fff8f171eaadc706272 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.719940 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9f662442aecd4fe5a7d273b0c1e6aca6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.720345 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f662442aecd4fe5a7d273b0c1e6aca6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.720378 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_34265ddfd85f4788b015e22efaf25d02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.720732 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34265ddfd85f4788b015e22efaf25d02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.718803 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5d7cf550840a4c5780ef193e148b71ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:50.725663 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2cfec52f9de94b2096a85e6c32337bcd" } modify_external_catalogs: true
I20251026 02:56:50.725809 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2cfec52f9de94b2096a85e6c32337bcd" } modify_external_catalogs: true
W20251026 02:56:50.725908 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d7cf550840a4c5780ef193e148b71ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.730317 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dbbfb77851d443f8cf4d798f3e07e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.734777 1661 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 93d8db69c285430687f5e99110f4bfad
I20251026 02:56:50.735941 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 93d8db69c285430687f5e99110f4bfad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.736558 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 93d8db69c285430687f5e99110f4bfad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.738607 1659 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3cf2d84ee15542b888e863fa40b143fb" } modify_external_catalogs: true
I20251026 02:56:50.738785 1659 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3cf2d84ee15542b888e863fa40b143fb" } modify_external_catalogs: true
I20251026 02:56:50.740309 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 93d8db69c285430687f5e99110f4bfad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.742748 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 027dd92fc5cf4c9e847de3e55e0dac1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.743778 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1053) took 4589 ms. Trace:
I20251026 02:56:50.743870 2372 rpcz_store.cc:276] 1026 02:56:46.154352 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.154398 (+ 46us) service_pool.cc:225] Handling call
1026 02:56:50.730848 (+4576450us) ts_tablet_manager.cc:1151] Deleting tablet 142043b5d558490c9399626c3764ff7e
1026 02:56:50.743627 (+ 12779us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.743757 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4576253}
W20251026 02:56:50.744303 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 142043b5d558490c9399626c3764ff7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 142043b5d558490c9399626c3764ff7e already in progress: creating tablet
I20251026 02:56:50.747960 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9c224d9efebb4e9ea1f3b2b2bb63335e" } new_table_name: "default.Table_ebfbe3fb80c64eeda4c4e24c93b0e139" modify_external_catalogs: true
I20251026 02:56:50.748181 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_73be634ceddf488ea518d690587f6cd0" } new_table_name: "default.Table_98707dd355d24935b65cd120909958d3" modify_external_catalogs: true
I20251026 02:56:50.749893 1624 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
/tmp/dist-test-taskiHIxsS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.150.61:37655
--webserver_interface=127.1.150.61
--webserver_port=34385
--builtin_ntp_servers=127.1.150.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.150.62:44525,127.1.150.61:37655,127.1.150.60:33729
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251026 02:56:50.752792 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_548905e23d3c442c928134d3e7a2bb03" } new_table_name: "default.Table_138bcec41e4644a59ec7750d16a61c0b" modify_external_catalogs: true
I20251026 02:56:50.753751 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_853ff6e087ac44d8b3ea6ecb653d0a6e" } modify_external_catalogs: true
I20251026 02:56:50.753887 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_853ff6e087ac44d8b3ea6ecb653d0a6e" } modify_external_catalogs: true
I20251026 02:56:50.753464 2381 tablet_bootstrap.cc:492] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:50.754586 2381 ts_tablet_manager.cc:1403] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.005s
I20251026 02:56:50.755241 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_94bf0678dbe746219b474992c5acde07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.755728 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94bf0678dbe746219b474992c5acde07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.757710 1980 raft_consensus.cc:3060] T 41a1442fffe54add93666e3c78a0cc58 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.761600 2376 ts_tablet_manager.cc:1929] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:50.761734 2376 log.cc:1199] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/3bcd7dec9c2149069e48dcf5e564595e
I20251026 02:56:50.762060 2376 ts_tablet_manager.cc:1950] T 3bcd7dec9c2149069e48dcf5e564595e P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:50.762517 2381 raft_consensus.cc:359] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.765731 2375 tablet_bootstrap.cc:492] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:50.765911 2375 ts_tablet_manager.cc:1403] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.001s
I20251026 02:56:50.766603 2375 raft_consensus.cc:359] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.766774 2375 raft_consensus.cc:385] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.766825 2375 raft_consensus.cc:740] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.766929 2381 raft_consensus.cc:385] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.767089 2381 raft_consensus.cc:740] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.767035 2375 consensus_queue.cc:260] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.767387 2381 consensus_queue.cc:260] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.770088 2375 ts_tablet_manager.cc:1434] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 02:56:50.773566 2381 ts_tablet_manager.cc:1434] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251026 02:56:50.777870 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1054) took 4615 ms. Trace:
I20251026 02:56:50.777983 2372 rpcz_store.cc:276] 1026 02:56:46.162426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.162492 (+ 66us) service_pool.cc:225] Handling call
1026 02:56:50.744629 (+4582137us) ts_tablet_manager.cc:1151] Deleting tablet 586bd6e40a7a4785b71b2a3d542e2411
1026 02:56:50.777713 (+ 33084us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.777846 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4581919}
W20251026 02:56:50.778445 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 586bd6e40a7a4785b71b2a3d542e2411 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 586bd6e40a7a4785b71b2a3d542e2411 already in progress: creating tablet
I20251026 02:56:50.779142 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1052) took 4624 ms. Trace:
I20251026 02:56:50.780962 2381 tablet_bootstrap.cc:492] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:50.780962 2375 tablet_bootstrap.cc:492] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:50.782680 2372 rpcz_store.cc:276] 1026 02:56:46.154242 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.164235 (+ 9993us) service_pool.cc:225] Handling call
1026 02:56:50.778794 (+4614559us) ts_tablet_manager.cc:1151] Deleting tablet 075f6ca8787943a09443863c544a1284
1026 02:56:50.778806 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.779122 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4613166}
W20251026 02:56:50.783097 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 075f6ca8787943a09443863c544a1284 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 075f6ca8787943a09443863c544a1284 already in progress: creating tablet
I20251026 02:56:50.784336 2375 tablet_bootstrap.cc:654] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.785651 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1060) took 4551 ms. Trace:
I20251026 02:56:50.785741 2372 rpcz_store.cc:276] 1026 02:56:46.233699 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.234761 (+ 1062us) service_pool.cc:225] Handling call
1026 02:56:50.785055 (+4550294us) ts_tablet_manager.cc:1151] Deleting tablet e4c149e8b7f9420c8dba56da2e21f3f9
1026 02:56:50.785067 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.785633 (+ 566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4545893}
I20251026 02:56:50.789979 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1022) took 3899 ms. Trace:
I20251026 02:56:50.791790 2376 rpcz_store.cc:276] 1026 02:56:46.890507 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.890599 (+ 92us) service_pool.cc:225] Handling call
1026 02:56:50.580681 (+3690082us) ts_tablet_manager.cc:1151] Deleting tablet 3bcd7dec9c2149069e48dcf5e564595e
1026 02:56:50.580723 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.739003 (+158280us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.761489 (+ 22486us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.789947 (+ 28458us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8064,"tablet-delete.queue_time_us":3689810}
I20251026 02:56:50.794081 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1061) took 4540 ms. Trace:
I20251026 02:56:50.794189 2372 rpcz_store.cc:276] 1026 02:56:46.253222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.253277 (+ 55us) service_pool.cc:225] Handling call
1026 02:56:50.793934 (+4540657us) ts_tablet_manager.cc:1151] Deleting tablet 7bef60e3a10843b4be2022ab4b64be68
1026 02:56:50.793947 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.794058 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4540442}
W20251026 02:56:50.794692 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 7bef60e3a10843b4be2022ab4b64be68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bef60e3a10843b4be2022ab4b64be68 already in progress: creating tablet
I20251026 02:56:50.794929 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1026) took 3720 ms. Trace:
I20251026 02:56:50.795047 2376 rpcz_store.cc:276] 1026 02:56:47.074680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.074746 (+ 66us) service_pool.cc:225] Handling call
1026 02:56:50.794766 (+3720020us) ts_tablet_manager.cc:1151] Deleting tablet f80b5c4c4e2d4c14a8a5292a05127bc1
1026 02:56:50.794781 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.794903 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3719808}
I20251026 02:56:50.794199 2375 tablet_bootstrap.cc:492] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:50.796577 2375 ts_tablet_manager.cc:1403] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20251026 02:56:50.797336 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1029) took 3497 ms. Trace:
I20251026 02:56:50.797449 2376 rpcz_store.cc:276] 1026 02:56:47.299968 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.317271 (+ 17303us) service_pool.cc:225] Handling call
1026 02:56:50.797173 (+3479902us) ts_tablet_manager.cc:1151] Deleting tablet 68ad3d4a2d484badb4b35884517b72f1
1026 02:56:50.797187 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.797312 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3466732}
I20251026 02:56:50.800581 2376 tablet_replica.cc:333] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:50.800877 2376 raft_consensus.cc:2243] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:50.801065 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 3bcd7dec9c2149069e48dcf5e564595e (table default.Table_7eb7fd8131cc4bce81b6cc09c5062fc0 [id=5cbfb8198a8947b3abd1c0801e92f100]) successfully deleted
W20251026 02:56:50.801332 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet f80b5c4c4e2d4c14a8a5292a05127bc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f80b5c4c4e2d4c14a8a5292a05127bc1 already in progress: creating tablet
W20251026 02:56:50.801512 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 68ad3d4a2d484badb4b35884517b72f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68ad3d4a2d484badb4b35884517b72f1 already in progress: creating tablet
I20251026 02:56:50.801725 2372 tablet_replica.cc:333] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:50.803982 2372 raft_consensus.cc:2243] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:50.804652 2376 raft_consensus.cc:2272] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.805796 2372 raft_consensus.cc:2272] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:50.809280 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e9353957d5a046fc9f7a7e4316ff606b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:50.809641 2372 ts_tablet_manager.cc:1916] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:50.809898 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9353957d5a046fc9f7a7e4316ff606b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:50.810745 1655 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df585f11274f4fd2852890aa2c53f0c0 consensus state has no leader
W20251026 02:56:50.811039 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 109073eb664d4090b7a2b786146e13fa consensus state has no leader
I20251026 02:56:50.811129 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_48fb5691d1674c79a437be2759a72637"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:50.811345 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet d85848daf97044e980060490a145400c
I20251026 02:56:50.811453 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d77bb0c25d55465c971fe532c05a5754"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.811591 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48fb5691d1674c79a437be2759a72637 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:50.811897 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d77bb0c25d55465c971fe532c05a5754 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.812338 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 842740d1b30a49769a2fa77a2813c1ae
W20251026 02:56:50.813073 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0b9626c8ed14636acdf0a286a709625 consensus state has no leader
I20251026 02:56:50.813315 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet d85848daf97044e980060490a145400c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.813416 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 4019c3563909495bbc76259b4153e892
I20251026 02:56:50.813457 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 842740d1b30a49769a2fa77a2813c1ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.804754 2375 raft_consensus.cc:359] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.813848 2375 raft_consensus.cc:385] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.814250 2375 raft_consensus.cc:740] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.814278 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 4019c3563909495bbc76259b4153e892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.814896 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet d85848daf97044e980060490a145400c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.815232 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 842740d1b30a49769a2fa77a2813c1ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.815675 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 4019c3563909495bbc76259b4153e892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.817348 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet d85848daf97044e980060490a145400c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.817570 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 842740d1b30a49769a2fa77a2813c1ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.818360 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 4019c3563909495bbc76259b4153e892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.821009 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 868dbbfce40c4dd6b36cc220966b1986 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:42 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.822130 2375 consensus_queue.cc:260] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.823645 2375 ts_tablet_manager.cc:1434] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20251026 02:56:50.823962 2375 tablet_bootstrap.cc:492] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:50.825719 2376 ts_tablet_manager.cc:1916] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:50.836519 2378 tablet_bootstrap.cc:492] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:50.836786 2378 ts_tablet_manager.cc:1403] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.304s user 0.024s sys 0.040s
I20251026 02:56:50.837778 2378 raft_consensus.cc:359] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.838020 2378 raft_consensus.cc:385] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.838143 2378 raft_consensus.cc:740] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.838425 2378 consensus_queue.cc:260] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.840020 2378 ts_tablet_manager.cc:1434] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251026 02:56:50.841205 2378 tablet_bootstrap.cc:492] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:50.842137 2375 tablet_bootstrap.cc:654] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.843001 2378 tablet_bootstrap.cc:654] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.845347 2381 tablet_bootstrap.cc:654] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:50.846338 2194 consensus_peers.cc:597] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:50.847847 2260 raft_consensus.cc:3060] T cfac07917ee34e6dba3fac5e081f6b43 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.851333 4098 consensus_queue.cc:1048] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:50.858479 2372 ts_tablet_manager.cc:1929] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.863751 2381 tablet_bootstrap.cc:492] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:50.864033 2381 ts_tablet_manager.cc:1403] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.083s user 0.000s sys 0.007s
I20251026 02:56:50.864955 2381 raft_consensus.cc:359] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.865283 2381 raft_consensus.cc:385] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.865391 2381 raft_consensus.cc:740] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.865700 2381 consensus_queue.cc:260] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.867898 4069 raft_consensus.cc:493] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.867975 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e961fa54c9e84ce0bc37f0aa04d6e4a1" } new_table_name: "default.Table_c5f164d2f87d4ed1a0ebaf7953d430d2" modify_external_catalogs: true
I20251026 02:56:50.868039 4069 raft_consensus.cc:515] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.868729 4069 leader_election.cc:290] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:50.868789 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_1103bb37accf4daf87c2bc9427de7d43" } new_table_name: "default.Table_78347c99c31b4334900a2ca386f15ca8" modify_external_catalogs: true
I20251026 02:56:50.869899 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85420a102dae4269811dbdc64aebb20a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:50.870677 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9c1813a4f5414f74a27525b8906a380b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.870934 2194 leader_election.cc:343] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:50.871186 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c1813a4f5414f74a27525b8906a380b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.871345 2381 ts_tablet_manager.cc:1434] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251026 02:56:50.875160 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c4c451a485f24525b3dbb3f3626a01e7" } modify_external_catalogs: true
I20251026 02:56:50.875355 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c4c451a485f24525b3dbb3f3626a01e7" } modify_external_catalogs: true
I20251026 02:56:50.875576 2372 log.cc:1199] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/209fa71af13142a58efb9f3dab3eecd3
I20251026 02:56:50.875981 2372 ts_tablet_manager.cc:1950] T 209fa71af13142a58efb9f3dab3eecd3 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:50.876735 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1062) took 4581 ms. Trace:
I20251026 02:56:50.876820 2372 rpcz_store.cc:276] 1026 02:56:46.295298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.296864 (+ 1566us) service_pool.cc:225] Handling call
1026 02:56:50.797585 (+4500721us) ts_tablet_manager.cc:1151] Deleting tablet 209fa71af13142a58efb9f3dab3eecd3
1026 02:56:50.797596 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.853473 (+ 55877us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.858378 (+ 4905us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.876716 (+ 18338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4500522}
I20251026 02:56:50.877234 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 209fa71af13142a58efb9f3dab3eecd3 (table default.Table_f4fc99282b5643e89641bb7082d165a5 [id=1e6385207c1648f8a87346a3ca603cb0]) successfully deleted
I20251026 02:56:50.859611 2375 tablet_bootstrap.cc:492] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:50.869223 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85420a102dae4269811dbdc64aebb20a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:50.877597 2375 ts_tablet_manager.cc:1403] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20251026 02:56:50.877602 2116 raft_consensus.cc:2468] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:50.877991 2195 leader_election.cc:304] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:50.878377 4069 raft_consensus.cc:2804] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:50.878471 4069 raft_consensus.cc:493] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:50.878528 4069 raft_consensus.cc:3060] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.878894 2375 raft_consensus.cc:359] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.879045 2375 raft_consensus.cc:385] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.879101 2375 raft_consensus.cc:740] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.879320 2375 consensus_queue.cc:260] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.880662 2375 ts_tablet_manager.cc:1434] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251026 02:56:50.881524 4069 raft_consensus.cc:515] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.881840 2377 ts_tablet_manager.cc:1929] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:50.881924 2377 log.cc:1199] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/ce1ab21ddf8e45bc9063b63db9c1b97f
I20251026 02:56:50.882226 2377 ts_tablet_manager.cc:1950] T ce1ab21ddf8e45bc9063b63db9c1b97f P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:50.882592 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85420a102dae4269811dbdc64aebb20a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:50.882921 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1111) took 3118 ms. Trace:
I20251026 02:56:50.882999 2377 rpcz_store.cc:276] 1026 02:56:47.764664 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.764714 (+ 50us) service_pool.cc:225] Handling call
1026 02:56:50.640272 (+2875558us) ts_tablet_manager.cc:1151] Deleting tablet ce1ab21ddf8e45bc9063b63db9c1b97f
1026 02:56:50.640283 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.878758 (+238475us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.881752 (+ 2994us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.882903 (+ 1151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2873330}
W20251026 02:56:50.883271 2194 leader_election.cc:343] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:50.883404 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet ce1ab21ddf8e45bc9063b63db9c1b97f (table default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5]) successfully deleted
I20251026 02:56:50.883708 4069 leader_election.cc:290] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:50.884127 2381 tablet_bootstrap.cc:492] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:50.885480 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85420a102dae4269811dbdc64aebb20a" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:50.885649 2116 raft_consensus.cc:3060] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.887356 2375 tablet_bootstrap.cc:492] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:50.889794 2116 raft_consensus.cc:2468] T 85420a102dae4269811dbdc64aebb20a P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:50.890233 2195 leader_election.cc:304] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:50.890173 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 76a093eaef6a48ba980ee1f745e4f25f (DEFAULT_TABLE table=default.Table_73c105436cb84727b8cabb2354aae56e [id=eebbff38179b4d8fb5eed1654e6789ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.890494 4069 raft_consensus.cc:2804] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:50.890583 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a093eaef6a48ba980ee1f745e4f25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.891371 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 9621b64b47a741acacd795bd4cb12dee (DEFAULT_TABLE table=default.Table_45834367050843509698000b4af518a5 [id=8db583a0bc6f4c8cbeff201332f328f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.891666 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9621b64b47a741acacd795bd4cb12dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.891774 4069 raft_consensus.cc:697] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:50.892113 4069 consensus_queue.cc:237] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.893564 4098 consensus_queue.cc:1048] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:50.898097 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 76a093eaef6a48ba980ee1f745e4f25f (DEFAULT_TABLE table=default.Table_73c105436cb84727b8cabb2354aae56e [id=eebbff38179b4d8fb5eed1654e6789ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.898440 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 9621b64b47a741acacd795bd4cb12dee (DEFAULT_TABLE table=default.Table_45834367050843509698000b4af518a5 [id=8db583a0bc6f4c8cbeff201332f328f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.898739 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 9621b64b47a741acacd795bd4cb12dee (DEFAULT_TABLE table=default.Table_45834367050843509698000b4af518a5 [id=8db583a0bc6f4c8cbeff201332f328f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.898834 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9621b64b47a741acacd795bd4cb12dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.902881 2375 tablet_bootstrap.cc:654] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.905930 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1067) took 4402 ms. Trace:
I20251026 02:56:50.907737 2372 rpcz_store.cc:276] 1026 02:56:46.503624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.528047 (+ 24423us) service_pool.cc:225] Handling call
1026 02:56:50.905806 (+4377759us) ts_tablet_manager.cc:1151] Deleting tablet a012d868a78f4324a92db83c81931b9d
1026 02:56:50.905817 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.905906 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4377527}
W20251026 02:56:50.908768 1648 catalog_manager.cc:4952] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet a012d868a78f4324a92db83c81931b9d because the tablet was not found. No further retry: Not found: Tablet not found: a012d868a78f4324a92db83c81931b9d
I20251026 02:56:50.908946 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 3aed4584b49a4eccac2ba44454866d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.909561 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1071) took 4378 ms. Trace:
I20251026 02:56:50.910737 2372 rpcz_store.cc:276] 1026 02:56:46.531202 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.536291 (+ 5089us) service_pool.cc:225] Handling call
1026 02:56:50.909442 (+4373151us) ts_tablet_manager.cc:1151] Deleting tablet 51a39824f5404d45b1756a2581aa1da4
1026 02:56:50.909451 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.909543 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4372709}
W20251026 02:56:50.912447 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 51a39824f5404d45b1756a2581aa1da4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51a39824f5404d45b1756a2581aa1da4 already in progress: creating tablet
I20251026 02:56:50.913425 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 5de04ef50ef241889f7a27a2825f6f7e (DEFAULT_TABLE table=default.Table_eb612503151340bf85e80ad83525e96d [id=7db28eeb33ba413ba2d64cee6f0f4da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.914186 2098 tablet_service.cc:1505] Processing CreateTablet for tablet b9fa749c17c847fda0150613df86a073 (DEFAULT_TABLE table=default.Table_773097afa38549ee9cf02a6349454c59 [id=82a854b5602f43ba8e430361e28468ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.912086 2381 tablet_bootstrap.cc:654] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:50.914904 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 45fd45ca857e45a097e86be7539f069c (DEFAULT_TABLE table=default.Table_738af4cbe3c5484ebb053c840d593c22 [id=19616ca557604b338706161265c1c6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.915359 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45fd45ca857e45a097e86be7539f069c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.918782 1655 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet cfac07917ee34e6dba3fac5e081f6b43 (table default.Table_907991c2b0034c118065ead079f273f9 [id=a8d490f3dbc948d38c54d14e178b736e]). Expected version 1 got 0
I20251026 02:56:50.919298 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 4992fad9bbc14ce2b206ac4373b05e5a (DEFAULT_TABLE table=default.Table_b8b1c6cdc277442094096a590903a0ee [id=2d595b2cd7aa419ab96c5317e2de94f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.919628 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4992fad9bbc14ce2b206ac4373b05e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.920440 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 70b1f8cf006148ec86de23d037ba5fc7 (DEFAULT_TABLE table=default.Table_9ad5fd6d3de34e97b6c8f52360294bb4 [id=ad5221f1c10f4d8e8e9063b0bfccefb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.920734 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70b1f8cf006148ec86de23d037ba5fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.924863 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 5de04ef50ef241889f7a27a2825f6f7e (DEFAULT_TABLE table=default.Table_eb612503151340bf85e80ad83525e96d [id=7db28eeb33ba413ba2d64cee6f0f4da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.926002 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 70b1f8cf006148ec86de23d037ba5fc7 (DEFAULT_TABLE table=default.Table_9ad5fd6d3de34e97b6c8f52360294bb4 [id=ad5221f1c10f4d8e8e9063b0bfccefb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.926088 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 45fd45ca857e45a097e86be7539f069c (DEFAULT_TABLE table=default.Table_738af4cbe3c5484ebb053c840d593c22 [id=19616ca557604b338706161265c1c6e9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:50.934392 1654 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91f81d156ff44decb63a4c0f5eacdac9 consensus state has no leader
I20251026 02:56:50.934893 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de04ef50ef241889f7a27a2825f6f7e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:50.935106 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6fa6400ae20a4b0d8a9a2013b3040ec6 consensus state has no leader
I20251026 02:56:50.938486 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2060154e868f41c3891ac2af97a423cc" } modify_external_catalogs: true
I20251026 02:56:50.938684 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2060154e868f41c3891ac2af97a423cc" } modify_external_catalogs: true
I20251026 02:56:50.939877 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9fa749c17c847fda0150613df86a073. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.942775 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 77d4998abe3a4a53a980b6d94b0021a9
I20251026 02:56:50.945304 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 77d4998abe3a4a53a980b6d94b0021a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.948383 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 77d4998abe3a4a53a980b6d94b0021a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.957386 1982 raft_consensus.cc:3060] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.958010 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 5de04ef50ef241889f7a27a2825f6f7e (DEFAULT_TABLE table=default.Table_eb612503151340bf85e80ad83525e96d [id=7db28eeb33ba413ba2d64cee6f0f4da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.958279 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 4992fad9bbc14ce2b206ac4373b05e5a (DEFAULT_TABLE table=default.Table_b8b1c6cdc277442094096a590903a0ee [id=2d595b2cd7aa419ab96c5317e2de94f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.958709 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 45fd45ca857e45a097e86be7539f069c (DEFAULT_TABLE table=default.Table_738af4cbe3c5484ebb053c840d593c22 [id=19616ca557604b338706161265c1c6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.958007 1954 tablet_service.cc:1505] Processing CreateTablet for tablet b9fa749c17c847fda0150613df86a073 (DEFAULT_TABLE table=default.Table_773097afa38549ee9cf02a6349454c59 [id=82a854b5602f43ba8e430361e28468ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.959307 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 70b1f8cf006148ec86de23d037ba5fc7 (DEFAULT_TABLE table=default.Table_9ad5fd6d3de34e97b6c8f52360294bb4 [id=ad5221f1c10f4d8e8e9063b0bfccefb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.958741 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 77d4998abe3a4a53a980b6d94b0021a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.961545 4069 raft_consensus.cc:493] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.961714 4069 raft_consensus.cc:515] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.962415 4069 leader_election.cc:290] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:50.963110 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fb42b54f558499090a1d7cf257dd84c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:50.963313 2116 raft_consensus.cc:2468] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:50.963753 2195 leader_election.cc:304] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:50.964053 4069 raft_consensus.cc:2804] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:50.964124 4069 raft_consensus.cc:493] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:50.964183 4069 raft_consensus.cc:3060] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.965389 4297 raft_consensus.cc:493] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:50.967500 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fb42b54f558499090a1d7cf257dd84c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:50.968324 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7834a85168364390a841fc883681c845
I20251026 02:56:50.970571 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 7834a85168364390a841fc883681c845 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:50.970697 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet 7834a85168364390a841fc883681c845 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:50.972165 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 7834a85168364390a841fc883681c845 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:50 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:50.965534 4297 raft_consensus.cc:515] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.972986 1975 raft_consensus.cc:2468] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:50.975252 2376 ts_tablet_manager.cc:1929] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:50.975344 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b52a73c60cda4bc58b800957c4f9bed1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:50.975845 2376 log.cc:1199] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/8c1e63ac8c5645f3ba1578792c0a0dae
W20251026 02:56:50.975958 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b52a73c60cda4bc58b800957c4f9bed1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.980543 4297 leader_election.cc:290] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:50.981072 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fb42b54f558499090a1d7cf257dd84c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:50.981264 2116 raft_consensus.cc:2468] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:50.981422 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fb42b54f558499090a1d7cf257dd84c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:50.981761 1916 leader_election.cc:304] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:50.982021 4302 raft_consensus.cc:2804] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:50.982087 4302 raft_consensus.cc:493] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:50.982141 4302 raft_consensus.cc:3060] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:50.982828 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a093eaef6a48ba980ee1f745e4f25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.983530 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45fd45ca857e45a097e86be7539f069c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.983810 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e6e2339629ed45d39e9070c3e05c109d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:50.984323 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4992fad9bbc14ce2b206ac4373b05e5a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:50.984365 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6e2339629ed45d39e9070c3e05c109d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.984841 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de04ef50ef241889f7a27a2825f6f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.985204 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9621b64b47a741acacd795bd4cb12dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.985528 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9520015608434b2990a378af62b56414"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.985975 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9520015608434b2990a378af62b56414 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:50.987470 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_9f6ae0bb97e0495895985e078c5e79b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:50.987911 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f6ae0bb97e0495895985e078c5e79b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:50.988200 1915 consensus_peers.cc:597] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Couldn't send request to peer 696476265495492790739ec126a08539. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f719c720cc474181a63786a3e248ab2c. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:50.988389 2372 tablet_replica.cc:333] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:50.988590 2372 raft_consensus.cc:2243] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:50.988754 2372 raft_consensus.cc:2272] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:50.990392 2311 tablet.cc:2378] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 02:56:50.990630 1915 consensus_peers.cc:597] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Couldn't send request to peer 696476265495492790739ec126a08539. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cf76ef8f6af445eb0ce15c8d79136c5. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:50.991411 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 4992fad9bbc14ce2b206ac4373b05e5a (DEFAULT_TABLE table=default.Table_b8b1c6cdc277442094096a590903a0ee [id=2d595b2cd7aa419ab96c5317e2de94f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.991825 2232 tablet_service.cc:1505] Processing CreateTablet for tablet b9fa749c17c847fda0150613df86a073 (DEFAULT_TABLE table=default.Table_773097afa38549ee9cf02a6349454c59 [id=82a854b5602f43ba8e430361e28468ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:50.992136 2375 tablet_bootstrap.cc:492] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:50.992271 2375 ts_tablet_manager.cc:1403] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.111s user 0.001s sys 0.003s
I20251026 02:56:50.992681 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de04ef50ef241889f7a27a2825f6f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.992977 2375 raft_consensus.cc:359] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.993134 2375 raft_consensus.cc:385] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:50.993170 2375 raft_consensus.cc:740] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:50.993229 2376 ts_tablet_manager.cc:1950] T 8c1e63ac8c5645f3ba1578792c0a0dae P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:50.993408 2375 consensus_queue.cc:260] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.993635 4069 raft_consensus.cc:515] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:50.993670 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1032) took 3612 ms. Trace:
I20251026 02:56:50.993934 2257 raft_consensus.cc:2393] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:50.993758 2376 rpcz_store.cc:276] 1026 02:56:47.380790 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.380842 (+ 52us) service_pool.cc:225] Handling call
1026 02:56:50.800528 (+3419686us) ts_tablet_manager.cc:1151] Deleting tablet 8c1e63ac8c5645f3ba1578792c0a0dae
1026 02:56:50.800537 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:50.857612 (+ 57075us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.974903 (+117291us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:50.993637 (+ 18734us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3408217}
I20251026 02:56:50.994531 2376 tablet_replica.cc:333] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:50.994869 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9fa749c17c847fda0150613df86a073. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.995568 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70b1f8cf006148ec86de23d037ba5fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.995815 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 8c1e63ac8c5645f3ba1578792c0a0dae (table default.Table_528af198a403496a8c1b27fa5821123b [id=e6aed02773494e3c8694deedf311adc6]) successfully deleted
I20251026 02:56:50.996599 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70b1f8cf006148ec86de23d037ba5fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:50.998661 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45fd45ca857e45a097e86be7539f069c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.001451 2376 raft_consensus.cc:2243] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:51.001883 2376 raft_consensus.cc:2272] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:51.002012 2233 tablet_service.cc:1505] Processing CreateTablet for tablet ac9abecec6a342488c3488276ceccbfc (DEFAULT_TABLE table=default.Table_cedb95f3248844edaeb1737e91c70c9b [id=a610a93b67a04c878cdf91571c197d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.002391 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9abecec6a342488c3488276ceccbfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.002436 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1136) took 2812 ms. Trace:
I20251026 02:56:51.002539 2377 rpcz_store.cc:276] 1026 02:56:48.189609 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.189671 (+ 62us) service_pool.cc:225] Handling call
1026 02:56:51.002298 (+2812627us) ts_tablet_manager.cc:1151] Deleting tablet 599a88ec0e76409c97ab62d6f55ebcd0
1026 02:56:51.002308 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.002418 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2811127}
I20251026 02:56:51.002823 2376 ts_tablet_manager.cc:1916] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:51.002930 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 599a88ec0e76409c97ab62d6f55ebcd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 599a88ec0e76409c97ab62d6f55ebcd0 already in progress: creating tablet
I20251026 02:56:51.005544 4302 raft_consensus.cc:515] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.006141 4302 leader_election.cc:290] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.007311 1661 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet cfac07917ee34e6dba3fac5e081f6b43 (table default.Table_907991c2b0034c118065ead079f273f9 [id=a8d490f3dbc948d38c54d14e178b736e]). Expected version 1 got 0
I20251026 02:56:51.008144 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fb42b54f558499090a1d7cf257dd84c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:51.008366 2116 raft_consensus.cc:3060] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.011066 2116 raft_consensus.cc:2468] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:51.012609 4069 leader_election.cc:290] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.013926 2375 ts_tablet_manager.cc:1434] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20251026 02:56:51.014443 2375 tablet_bootstrap.cc:492] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:51.016364 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fb42b54f558499090a1d7cf257dd84c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:51.016453 2375 tablet_bootstrap.cc:654] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:51.016597 1975 raft_consensus.cc:2393] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 696476265495492790739ec126a08539 in current term 1: Already voted for candidate 7fef01f4f2b843538ea245a09dd087a8 in this term.
I20251026 02:56:51.016670 1955 tablet_service.cc:1505] Processing CreateTablet for tablet ac9abecec6a342488c3488276ceccbfc (DEFAULT_TABLE table=default.Table_cedb95f3248844edaeb1737e91c70c9b [id=a610a93b67a04c878cdf91571c197d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.016999 1916 leader_election.cc:304] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8; no voters:
I20251026 02:56:51.020152 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fb42b54f558499090a1d7cf257dd84c" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:51.020424 2257 raft_consensus.cc:2393] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:51.021065 4069 consensus_queue.cc:1048] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:51.021659 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fb42b54f558499090a1d7cf257dd84c" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:51.021880 2116 raft_consensus.cc:2393] T 9fb42b54f558499090a1d7cf257dd84c P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 696476265495492790739ec126a08539 in current term 1: Already voted for candidate 7fef01f4f2b843538ea245a09dd087a8 in this term.
I20251026 02:56:51.022217 2195 leader_election.cc:304] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:51.022472 4302 raft_consensus.cc:2804] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:51.022571 4069 raft_consensus.cc:2749] T 9fb42b54f558499090a1d7cf257dd84c P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 02:56:51.026579 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e05eada8742471f87d73f5d960817c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:51.027652 2081 tablet_service.cc:1505] Processing CreateTablet for tablet ac9abecec6a342488c3488276ceccbfc (DEFAULT_TABLE table=default.Table_cedb95f3248844edaeb1737e91c70c9b [id=a610a93b67a04c878cdf91571c197d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.028126 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9abecec6a342488c3488276ceccbfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.029712 2372 ts_tablet_manager.cc:1916] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:51.032137 2375 tablet_bootstrap.cc:492] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:51.032302 2375 ts_tablet_manager.cc:1403] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.002s
I20251026 02:56:51.038396 2381 tablet_bootstrap.cc:492] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:51.038960 2381 ts_tablet_manager.cc:1403] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.155s user 0.005s sys 0.000s
I20251026 02:56:51.038676 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_6cb0771a15564fff8f171eaadc706272" } new_table_name: "default.Table_02cc9b3dbd8b4d43b6948f7b9eb3e566" modify_external_catalogs: true
I20251026 02:56:51.039003 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_8300040c56d34cf985a2009545c1eeb5" } new_table_name: "default.Table_c4cea0beb9da4527acd5a3aaebe14ce2" modify_external_catalogs: true
I20251026 02:56:51.039769 2381 raft_consensus.cc:359] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.039923 2381 raft_consensus.cc:385] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:51.039981 2381 raft_consensus.cc:740] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:51.040184 2381 consensus_queue.cc:260] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.040709 2381 ts_tablet_manager.cc:1434] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:51.040719 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_05ee3e62f67e424989346a2bd6f8389b" } modify_external_catalogs: true
I20251026 02:56:51.043660 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_05ee3e62f67e424989346a2bd6f8389b" } modify_external_catalogs: true
I20251026 02:56:51.046329 4255 raft_consensus.cc:493] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.046468 4255 raft_consensus.cc:515] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.047024 4255 leader_election.cc:290] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.047523 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109073eb664d4090b7a2b786146e13fa" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:51.047708 2110 raft_consensus.cc:2468] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:51.047943 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109073eb664d4090b7a2b786146e13fa" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:51.048440 2194 leader_election.cc:343] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.048578 2194 leader_election.cc:304] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:51.048902 4255 raft_consensus.cc:2804] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:51.048969 2381 tablet_bootstrap.cc:492] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:51.048971 4255 raft_consensus.cc:493] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:51.049281 4255 raft_consensus.cc:3060] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.050272 2375 raft_consensus.cc:359] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.050451 2375 raft_consensus.cc:385] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:51.050501 2375 raft_consensus.cc:740] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:51.050710 2375 consensus_queue.cc:260] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.051873 4255 raft_consensus.cc:515] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.054574 4255 leader_election.cc:290] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.055034 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109073eb664d4090b7a2b786146e13fa" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:51.055575 2194 leader_election.cc:343] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.056677 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9abecec6a342488c3488276ceccbfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.057675 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "109073eb664d4090b7a2b786146e13fa" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:51.057855 2110 raft_consensus.cc:3060] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.060034 2375 ts_tablet_manager.cc:1434] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20251026 02:56:51.060443 2375 tablet_bootstrap.cc:492] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:51.060976 2110 raft_consensus.cc:2468] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:51.061615 2195 leader_election.cc:304] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:51.062772 4069 raft_consensus.cc:2804] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:51.062891 4069 raft_consensus.cc:697] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:51.063112 4069 consensus_queue.cc:237] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.063987 4258 raft_consensus.cc:493] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.064090 4258 raft_consensus.cc:515] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.064656 4258 leader_election.cc:290] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.065706 2381 tablet_bootstrap.cc:654] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:51.073957 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9fa749c17c847fda0150613df86a073. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.074225 2381 tablet_bootstrap.cc:492] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:51.074394 2381 ts_tablet_manager.cc:1403] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.002s
I20251026 02:56:51.076449 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4992fad9bbc14ce2b206ac4373b05e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.077404 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1139) took 2887 ms. Trace:
I20251026 02:56:51.077481 2377 rpcz_store.cc:276] 1026 02:56:48.190070 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.191198 (+ 1128us) service_pool.cc:225] Handling call
1026 02:56:51.062191 (+2870993us) ts_tablet_manager.cc:1151] Deleting tablet 086b40036f0c4c97bb95647f91ce7939
1026 02:56:51.077268 (+ 15077us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.077384 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2870843}
I20251026 02:56:51.077780 2377 tablet_replica.cc:333] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:51.078003 2377 raft_consensus.cc:2243] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:51.078390 2377 raft_consensus.cc:2272] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:51.079007 2377 ts_tablet_manager.cc:1916] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:51.077790 2381 raft_consensus.cc:359] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.079391 2381 raft_consensus.cc:385] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:51.079439 2381 raft_consensus.cc:740] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:51.079625 2381 consensus_queue.cc:260] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.081498 2381 ts_tablet_manager.cc:1434] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251026 02:56:51.083240 2381 tablet_bootstrap.cc:492] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:51.089187 2381 tablet_bootstrap.cc:654] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:51.095713 4297 raft_consensus.cc:493] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 696476265495492790739ec126a08539)
I20251026 02:56:51.095858 4297 raft_consensus.cc:515] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.096642 4297 leader_election.cc:290] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.097666 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eabfc95caa84afc9a8956a7ac13fea6" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:51.099870 4302 raft_consensus.cc:697] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:51.100384 4302 consensus_queue.cc:237] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.102111 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dbbfb77851d443f8cf4d798f3e07e59" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.104974 4098 raft_consensus.cc:493] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.105114 4098 raft_consensus.cc:515] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.112721 2378 tablet_bootstrap.cc:492] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:51.112887 2378 ts_tablet_manager.cc:1403] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.272s user 0.010s sys 0.015s
I20251026 02:56:51.113682 2378 raft_consensus.cc:359] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.113835 2378 raft_consensus.cc:385] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:51.113884 2378 raft_consensus.cc:740] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:51.114032 2378 consensus_queue.cc:260] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.115658 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 70d9e78abcec4a609cf20b4f79730366 (DEFAULT_TABLE table=default.Table_dbd6c27f936b44f48242fff9d1b6ec82 [id=52bc3d0ac5ac4804bae712733f21929f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.116319 2232 tablet_service.cc:1505] Processing CreateTablet for tablet d9738312b038441ea3beec0bfb54f70c (DEFAULT_TABLE table=default.Table_98834edef3e047759f8740f45604592c [id=aab8db9070af46d1be5625bf9997d2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.116868 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 6fa6400ae20a4b0d8a9a2013b3040ec6 (DEFAULT_TABLE table=default.Table_c5f164d2f87d4ed1a0ebaf7953d430d2 [id=a78357bac5e14686a7f72504dbdc168b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.116987 2235 tablet_service.cc:1505] Processing CreateTablet for tablet b399723adb5d4ac0a842ebed6c2e029c (DEFAULT_TABLE table=default.Table_9615d980ff294c7f9cee5fd619547a01 [id=ebf32ef4833246b4abaff4b8477e25cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.117357 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70d9e78abcec4a609cf20b4f79730366. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.117741 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fa6400ae20a4b0d8a9a2013b3040ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.118188 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b399723adb5d4ac0a842ebed6c2e029c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.124327 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9738312b038441ea3beec0bfb54f70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.124249 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d89900bf7bf04855a2bd970460c420e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.124853 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d89900bf7bf04855a2bd970460c420e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.124943 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_c6376c7fc54d43c68859ff371a420894" } new_table_name: "default.Table_445bab59b72c4ea887a461fe60950f44" modify_external_catalogs: true
I20251026 02:56:51.125818 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet cfac07917ee34e6dba3fac5e081f6b43 (table default.Table_907991c2b0034c118065ead079f273f9 [id=a8d490f3dbc948d38c54d14e178b736e]). Expected version 1 got 0
I20251026 02:56:51.125988 1658 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 109073eb664d4090b7a2b786146e13fa (table default.Table_ebfbe3fb80c64eeda4c4e24c93b0e139 [id=da72d631aee04d10a6832b04f208587f]). Expected version 5 got 0
W20251026 02:56:51.141294 2168 tablet.cc:2378] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 02:56:51.142122 2098 tablet_service.cc:1505] Processing CreateTablet for tablet d9738312b038441ea3beec0bfb54f70c (DEFAULT_TABLE table=default.Table_98834edef3e047759f8740f45604592c [id=aab8db9070af46d1be5625bf9997d2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.142282 4098 leader_election.cc:290] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.142551 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9738312b038441ea3beec0bfb54f70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.143242 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e05eada8742471f87d73f5d960817c0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:51.143760 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 70d9e78abcec4a609cf20b4f79730366 (DEFAULT_TABLE table=default.Table_dbd6c27f936b44f48242fff9d1b6ec82 [id=52bc3d0ac5ac4804bae712733f21929f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.144140 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70d9e78abcec4a609cf20b4f79730366. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:51.144518 2053 leader_election.cc:343] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.152951 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 70d9e78abcec4a609cf20b4f79730366 (DEFAULT_TABLE table=default.Table_dbd6c27f936b44f48242fff9d1b6ec82 [id=52bc3d0ac5ac4804bae712733f21929f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.153693 2378 ts_tablet_manager.cc:1434] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.041s user 0.001s sys 0.000s
I20251026 02:56:51.066412 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dbbfb77851d443f8cf4d798f3e07e59" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:51.154204 2084 tablet_service.cc:1505] Processing CreateTablet for tablet 6fa6400ae20a4b0d8a9a2013b3040ec6 (DEFAULT_TABLE table=default.Table_c5f164d2f87d4ed1a0ebaf7953d430d2 [id=a78357bac5e14686a7f72504dbdc168b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.154579 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fa6400ae20a4b0d8a9a2013b3040ec6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:51.158135 2195 leader_election.cc:343] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Illegal state: Tablet not RUNNING: STOPPED
I20251026 02:56:51.158257 2195 leader_election.cc:304] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:51.158700 4069 raft_consensus.cc:2749] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:51.158634 1656 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_738af4cbe3c5484ebb053c840d593c22" } modify_external_catalogs: true
I20251026 02:56:51.158804 1656 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_738af4cbe3c5484ebb053c840d593c22" } modify_external_catalogs: true
I20251026 02:56:51.159745 3997 raft_consensus.cc:493] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.159878 3997 raft_consensus.cc:515] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.160404 3997 leader_election.cc:290] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.161661 2381 tablet_bootstrap.cc:492] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:51.161782 2381 ts_tablet_manager.cc:1403] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.079s user 0.017s sys 0.019s
W20251026 02:56:51.164435 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d8c9c0792f944758fbaa658a99f1c71 consensus state has no leader
I20251026 02:56:51.164849 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_031fab6bd5b5488fb13986995c2dd70c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.165474 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_031fab6bd5b5488fb13986995c2dd70c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.166777 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_35d854aa640a4cddbbbada2647659531"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.167207 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35d854aa640a4cddbbbada2647659531 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.168475 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_dcb798e05db24976a7f83acce835a73a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.168852 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcb798e05db24976a7f83acce835a73a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.174168 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5800c367f03d4a7fb9475a364d30e14b" } modify_external_catalogs: true
I20251026 02:56:51.174458 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5800c367f03d4a7fb9475a364d30e14b" } modify_external_catalogs: true
I20251026 02:56:51.176893 1663 catalog_manager.cc:5649] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:51.177886 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 109073eb664d4090b7a2b786146e13fa (table default.Table_ebfbe3fb80c64eeda4c4e24c93b0e139 [id=da72d631aee04d10a6832b04f208587f]). Expected version 5 got 0
I20251026 02:56:51.178257 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 41a1442fffe54add93666e3c78a0cc58 (table default.Table_1faa0c8151724c5e816546ee2e1d5182 [id=a106f9ece0b9467a8bc1d5d4b3524c41]). Expected version 1 got 0
I20251026 02:56:51.178821 1663 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 398c1f420974453b9b6b0fe17795378b (table default.Table_a92e68d05f4b455d810180ee4cc5874d [id=7f64fbec3582415a8924d01e14856a56]). Expected version 1 got 0
W20251026 02:56:51.181241 1647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.60:33729, user_credentials={real_user=slave}} blocked reactor thread for 59278.8us
I20251026 02:56:51.181954 4376 raft_consensus.cc:493] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.182101 4376 raft_consensus.cc:515] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.188972 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 62f7053bfff54e8ea3631a264431d80d (DEFAULT_TABLE table=default.Table_0aab08dd62de4321ba16eae6624f2d5b [id=7e000cab14184e2881cd6f9ba58522e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.189689 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 790a6b5db36c4fb2a83e8a6a43e5414b (DEFAULT_TABLE table=default.Table_8d7bf2ec1d04457cba728889928fae0c [id=c327ab2a7d054c689020fc1ed90acbae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.189848 2084 tablet_service.cc:1505] Processing CreateTablet for tablet b399723adb5d4ac0a842ebed6c2e029c (DEFAULT_TABLE table=default.Table_9615d980ff294c7f9cee5fd619547a01 [id=ebf32ef4833246b4abaff4b8477e25cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.189692 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f7053bfff54e8ea3631a264431d80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.190623 2081 tablet_service.cc:1505] Processing CreateTablet for tablet 0a7b81b4ed2c49bfa1674d5775110545 (DEFAULT_TABLE table=default.Table_79d8aa03b0ed42feab9dd85971ae173b [id=574a4603b9d8478dae2092fd2b7bd3bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.190949 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a7b81b4ed2c49bfa1674d5775110545. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.193358 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 790a6b5db36c4fb2a83e8a6a43e5414b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.193393 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 9067573953fb4e2bb13d979bd4742dc4 (DEFAULT_TABLE table=default.Table_fc36553c686349288efdaf7986cd5302 [id=c17b1d615de64c1b974f51620dacda9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.193640 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 6bc01d5ee1284036b3c70d07c5e763e7 (DEFAULT_TABLE table=default.Table_272aa94f33a74e0f99feb2cb36453493 [id=b30d6e7092304ad5986e6a19cbbeae37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.194135 2100 tablet_service.cc:1505] Processing CreateTablet for tablet d1ace19174ec4cca8314816665583cb2 (DEFAULT_TABLE table=default.Table_d744c8c07d3c4f25b019bfe49bf82b68 [id=1797a88a396d4209ba823205e75fc294]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.193663 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9067573953fb4e2bb13d979bd4742dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.194880 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 62f7053bfff54e8ea3631a264431d80d (DEFAULT_TABLE table=default.Table_0aab08dd62de4321ba16eae6624f2d5b [id=7e000cab14184e2881cd6f9ba58522e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.195217 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f7053bfff54e8ea3631a264431d80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.196453 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 9067573953fb4e2bb13d979bd4742dc4 (DEFAULT_TABLE table=default.Table_fc36553c686349288efdaf7986cd5302 [id=c17b1d615de64c1b974f51620dacda9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.196758 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9067573953fb4e2bb13d979bd4742dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.197357 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bc01d5ee1284036b3c70d07c5e763e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.197765 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1ace19174ec4cca8314816665583cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.198233 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 0a7b81b4ed2c49bfa1674d5775110545 (DEFAULT_TABLE table=default.Table_79d8aa03b0ed42feab9dd85971ae173b [id=574a4603b9d8478dae2092fd2b7bd3bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.198552 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a7b81b4ed2c49bfa1674d5775110545. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.200438 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 790a6b5db36c4fb2a83e8a6a43e5414b (DEFAULT_TABLE table=default.Table_8d7bf2ec1d04457cba728889928fae0c [id=c327ab2a7d054c689020fc1ed90acbae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.200733 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 790a6b5db36c4fb2a83e8a6a43e5414b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:51.203349 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b96528939e4445e09dec2195b4302a3e has no consensus state
I20251026 02:56:51.203649 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d1d2850518cc4c51a5cd3296d2d19680" } new_table_name: "default.Table_039d5df7cf8245a48168ef2ba4864e0f" modify_external_catalogs: true
I20251026 02:56:51.203696 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_cd4901edb6854b86b0076a24935350ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:51.204420 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 6bc01d5ee1284036b3c70d07c5e763e7 (DEFAULT_TABLE table=default.Table_272aa94f33a74e0f99feb2cb36453493 [id=b30d6e7092304ad5986e6a19cbbeae37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.204797 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bc01d5ee1284036b3c70d07c5e763e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.204999 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet ceb7249968ff44c1b5be1d1d18605ffe
I20251026 02:56:51.205783 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b399723adb5d4ac0a842ebed6c2e029c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.205767 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_31f69698add24de2aaccc829dd116c38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.206239 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31f69698add24de2aaccc829dd116c38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.207023 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet ceb7249968ff44c1b5be1d1d18605ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:51.207422 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ca8189991574d7c9d432dd737fee13e consensus state has no leader
I20251026 02:56:51.207690 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_debeb38e035a435fa936ab479b12d30e" } new_table_name: "default.Table_591a2f198bfe45d1b90f60a732efc1e4" modify_external_catalogs: true
I20251026 02:56:51.208513 2241 tablet_service.cc:1505] Processing CreateTablet for tablet d1ace19174ec4cca8314816665583cb2 (DEFAULT_TABLE table=default.Table_d744c8c07d3c4f25b019bfe49bf82b68 [id=1797a88a396d4209ba823205e75fc294]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.208644 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7b456a8685d84372a3b4710e3ad99da5" } modify_external_catalogs: true
I20251026 02:56:51.208765 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7b456a8685d84372a3b4710e3ad99da5" } modify_external_catalogs: true
I20251026 02:56:51.208848 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1ace19174ec4cca8314816665583cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.210191 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_645745dc5d2840adaede1a3708978a74" } modify_external_catalogs: true
I20251026 02:56:51.210263 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1dd78f088f84a509c761c934c039016" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.210427 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_645745dc5d2840adaede1a3708978a74" } modify_external_catalogs: true
I20251026 02:56:51.210748 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e05eada8742471f87d73f5d960817c0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.211925 1656 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 45fd45ca857e45a097e86be7539f069c
I20251026 02:56:51.212867 2375 tablet_bootstrap.cc:654] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:51.214762 1916 leader_election.cc:343] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 6ea364c4a5cf4f3f99d9ff6904add697
W20251026 02:56:51.204185 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd4901edb6854b86b0076a24935350ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.216488 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70d9e78abcec4a609cf20b4f79730366. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.216889 4305 raft_consensus.cc:493] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.216913 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 11f4c327caa644d89098496bc324b976
I20251026 02:56:51.217010 4305 raft_consensus.cc:515] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.217777 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 45fd45ca857e45a097e86be7539f069c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:51.217952 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f4c327caa644d89098496bc324b976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:51.213291 4376 leader_election.cc:290] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.223225 2376 ts_tablet_manager.cc:1929] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:51.223361 2376 log.cc:1199] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/f719c720cc474181a63786a3e248ab2c
I20251026 02:56:51.224637 3997 raft_consensus.cc:493] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.224774 3997 raft_consensus.cc:515] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.224967 2372 ts_tablet_manager.cc:1929] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:51.225406 3997 leader_election.cc:290] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:51.225528 2030 tablet.cc:2378] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 02:56:51.225723 1955 tablet_service.cc:1505] Processing CreateTablet for tablet d9738312b038441ea3beec0bfb54f70c (DEFAULT_TABLE table=default.Table_98834edef3e047759f8740f45604592c [id=aab8db9070af46d1be5625bf9997d2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.226837 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "027dd92fc5cf4c9e847de3e55e0dac1c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:51.227324 2053 leader_election.cc:343] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.227267 2372 log.cc:1199] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/1dbbfb77851d443f8cf4d798f3e07e59
I20251026 02:56:51.228586 4305 leader_election.cc:290] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.160866 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1dd78f088f84a509c761c934c039016" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:51.229442 1954 tablet_service.cc:1505] Processing CreateTablet for tablet b399723adb5d4ac0a842ebed6c2e029c (DEFAULT_TABLE table=default.Table_9615d980ff294c7f9cee5fd619547a01 [id=ebf32ef4833246b4abaff4b8477e25cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.229559 2381 raft_consensus.cc:359] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.229696 2381 raft_consensus.cc:385] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:51.229734 2381 raft_consensus.cc:740] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:51.229905 2381 consensus_queue.cc:260] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.230345 2381 ts_tablet_manager.cc:1434] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.068s user 0.001s sys 0.001s
I20251026 02:56:51.231231 2375 tablet_bootstrap.cc:492] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:51.231848 2381 tablet_bootstrap.cc:492] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:51.232384 2375 ts_tablet_manager.cc:1403] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.172s user 0.006s sys 0.000s
W20251026 02:56:51.232566 2053 leader_election.cc:343] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.232688 2372 ts_tablet_manager.cc:1950] T 1dbbfb77851d443f8cf4d798f3e07e59 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:51.233685 2376 ts_tablet_manager.cc:1950] T f719c720cc474181a63786a3e248ab2c P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:51.242565 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1066) took 4739 ms. Trace:
I20251026 02:56:51.242827 2375 raft_consensus.cc:359] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.244182 2375 raft_consensus.cc:385] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:51.243474 2372 rpcz_store.cc:276] 1026 02:56:46.503183 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.539033 (+ 35850us) service_pool.cc:225] Handling call
1026 02:56:50.912848 (+4373815us) ts_tablet_manager.cc:1151] Deleting tablet 1dbbfb77851d443f8cf4d798f3e07e59
1026 02:56:50.988335 (+ 75487us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.163404 (+175069us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.209706 (+ 46302us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.242340 (+ 32634us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4373451}
I20251026 02:56:51.244763 2375 raft_consensus.cc:740] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:51.245296 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 1dbbfb77851d443f8cf4d798f3e07e59 (table default.Table_065759ae67c2414ca2b73a3f68a4111c [id=b7f8d9dfa47b4c499af7d0e09d417c9b]) successfully deleted
I20251026 02:56:51.248365 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1031) took 3853 ms. Trace:
I20251026 02:56:51.248504 2376 rpcz_store.cc:276] 1026 02:56:47.380647 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.380867 (+ 220us) service_pool.cc:225] Handling call
1026 02:56:50.994470 (+3613603us) ts_tablet_manager.cc:1151] Deleting tablet f719c720cc474181a63786a3e248ab2c
1026 02:56:50.994479 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.219359 (+224880us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.223137 (+ 3778us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.233998 (+ 10861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3602141}
I20251026 02:56:51.251765 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 790a6b5db36c4fb2a83e8a6a43e5414b (DEFAULT_TABLE table=default.Table_8d7bf2ec1d04457cba728889928fae0c [id=c327ab2a7d054c689020fc1ed90acbae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.252249 2378 tablet_bootstrap.cc:492] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:51.253166 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet ceb7249968ff44c1b5be1d1d18605ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:51.259428 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 45fd45ca857e45a097e86be7539f069c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:51.260025 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f4c327caa644d89098496bc324b976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:51.261325 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 76c779bae091430fac46ad5dffb360ca (DEFAULT_TABLE table=default.Table_8e0fc984298540898b1ae289f95d504f [id=90b00dc4d553430c8b63dccd1a81ebd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.261922 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c779bae091430fac46ad5dffb360ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.265197 2100 tablet_service.cc:1505] Processing CreateTablet for tablet ffe8d17204cb40a09992b5600958e64e (DEFAULT_TABLE table=default.Table_55b76386af9a4934813a5c92774f5697 [id=89d5f588892a42498df49590ec6e2f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.265712 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe8d17204cb40a09992b5600958e64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.266405 2377 ts_tablet_manager.cc:1929] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:51.267378 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 6fa6400ae20a4b0d8a9a2013b3040ec6 (DEFAULT_TABLE table=default.Table_c5f164d2f87d4ed1a0ebaf7953d430d2 [id=a78357bac5e14686a7f72504dbdc168b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:51.267836 1916 leader_election.cc:343] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9b9a5daa36b547a7888d38dc3329eaf2
I20251026 02:56:51.268471 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9738312b038441ea3beec0bfb54f70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.272255 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 62f7053bfff54e8ea3631a264431d80d (DEFAULT_TABLE table=default.Table_0aab08dd62de4321ba16eae6624f2d5b [id=7e000cab14184e2881cd6f9ba58522e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.278779 2082 tablet_service.cc:1505] Processing CreateTablet for tablet ee46aae2e3304a3ea7ac5e02ea4b3b4b (DEFAULT_TABLE table=default.Table_9fef71477b964049a34fada596af80bf [id=240bb6da01cf4c1b98f9301e17cc7817]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.278829 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 047bfb3525054c44aeb7f37625d9d8b8 (DEFAULT_TABLE table=default.Table_7e72edb3045a47d1ab44c37eadf3c705 [id=140fa200b8934fd1b0f8098b27dd9a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.279215 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee46aae2e3304a3ea7ac5e02ea4b3b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.277350 2381 tablet_bootstrap.cc:654] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:51.285413 4258 raft_consensus.cc:493] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.285571 4258 raft_consensus.cc:515] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.285611 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1033) took 3851 ms. Trace:
I20251026 02:56:51.285845 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 0a7b81b4ed2c49bfa1674d5775110545 (DEFAULT_TABLE table=default.Table_79d8aa03b0ed42feab9dd85971ae173b [id=574a4603b9d8478dae2092fd2b7bd3bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.286157 4258 leader_election.cc:290] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.286556 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66ede86c604e47dbb982e5a3ba18bcb7" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:51.286723 2110 raft_consensus.cc:2468] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:51.287181 2195 leader_election.cc:304] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:51.287231 2376 rpcz_store.cc:276] 1026 02:56:47.397195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.397249 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:51.248755 (+3851506us) ts_tablet_manager.cc:1151] Deleting tablet 0aabfe11dedd454f8221fb5de25a268b
1026 02:56:51.248765 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.248866 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3851349}
I20251026 02:56:51.287451 4258 raft_consensus.cc:2804] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:51.287529 4258 raft_consensus.cc:493] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:51.287599 4258 raft_consensus.cc:3060] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.287591 2376 tablet_replica.cc:333] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:51.289242 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 36fca225e913492e84a96043fa7ed215 (DEFAULT_TABLE table=default.Table_67618fda8a6d4b7a85112555b0961143 [id=cdfb39a67e904cef9266733ac9848c0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.289631 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fca225e913492e84a96043fa7ed215. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.290637 4258 raft_consensus.cc:515] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.291651 4258 leader_election.cc:290] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.292068 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66ede86c604e47dbb982e5a3ba18bcb7" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:51.292270 2110 raft_consensus.cc:3060] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.294453 2377 log.cc:1199] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/5a00bc04b3654f90af342986751344e6
I20251026 02:56:51.294874 2377 ts_tablet_manager.cc:1950] T 5a00bc04b3654f90af342986751344e6 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:51.294982 2110 raft_consensus.cc:2468] T 66ede86c604e47dbb982e5a3ba18bcb7 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:51.295694 2195 leader_election.cc:304] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:51.295755 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1137) took 3106 ms. Trace:
I20251026 02:56:51.295987 4258 raft_consensus.cc:2804] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:51.296322 4258 raft_consensus.cc:697] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:51.296665 4258 consensus_queue.cc:237] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.298424 2087 tablet_service.cc:1505] Processing CreateTablet for tablet 01338cc196084f38b31bf5c41751ece1 (DEFAULT_TABLE table=default.Table_c3b49514cb69429ca8b6cc1fd3422ddf [id=a966e7a92b454d419fb12dab06067b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.298897 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01338cc196084f38b31bf5c41751ece1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.299263 3997 raft_consensus.cc:493] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.299427 3997 raft_consensus.cc:515] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.300125 3997 leader_election.cc:290] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.300536 2099 tablet_service.cc:1505] Processing CreateTablet for tablet 53e4ac96055a4de89485f3eeb5e1fcf2 (DEFAULT_TABLE table=default.Table_a09f60f754244ce3958f5633cf37ec42 [id=7036cf643bdf4e39a541d3c7ec37b53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.300889 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53e4ac96055a4de89485f3eeb5e1fcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.309019 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aed4584b49a4eccac2ba44454866d49" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:51.310070 2053 leader_election.cc:343] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.310622 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f7053bfff54e8ea3631a264431d80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.295850 2377 rpcz_store.cc:276] 1026 02:56:48.189721 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.189767 (+ 46us) service_pool.cc:225] Handling call
1026 02:56:51.077740 (+2887973us) ts_tablet_manager.cc:1151] Deleting tablet 5a00bc04b3654f90af342986751344e6
1026 02:56:51.077749 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.150689 (+ 72940us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.266293 (+115604us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.295732 (+ 29439us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2886336}
I20251026 02:56:51.312412 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1138) took 3122 ms. Trace:
I20251026 02:56:51.312490 2377 rpcz_store.cc:276] 1026 02:56:48.189816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.189858 (+ 42us) service_pool.cc:225] Handling call
1026 02:56:51.312233 (+3122375us) ts_tablet_manager.cc:1151] Deleting tablet c0e6905e5ac147e3bc3a0073816faf76
1026 02:56:51.312293 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.312391 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3120780}
I20251026 02:56:51.312655 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 9067573953fb4e2bb13d979bd4742dc4 (DEFAULT_TABLE table=default.Table_fc36553c686349288efdaf7986cd5302 [id=c17b1d615de64c1b974f51620dacda9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.312760 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1140) took 3025 ms. Trace:
I20251026 02:56:51.312944 2377 rpcz_store.cc:276] 1026 02:56:48.287718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.287777 (+ 59us) service_pool.cc:225] Handling call
1026 02:56:51.312677 (+3024900us) ts_tablet_manager.cc:1151] Deleting tablet 8869062eedda4bfe99b1793ae0353680
1026 02:56:51.312685 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.312748 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3024754}
I20251026 02:56:51.313066 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9067573953fb4e2bb13d979bd4742dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.313371 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1141) took 3002 ms. Trace:
I20251026 02:56:51.313452 2377 rpcz_store.cc:276] 1026 02:56:48.310614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.310665 (+ 51us) service_pool.cc:225] Handling call
1026 02:56:51.313270 (+3002605us) ts_tablet_manager.cc:1151] Deleting tablet 4ed205c1800243ed9830d720ac85de2f
1026 02:56:51.313292 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.313357 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3002366}
I20251026 02:56:51.313863 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1148) took 2436 ms. Trace:
I20251026 02:56:51.313931 2377 rpcz_store.cc:276] 1026 02:56:48.877276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.877347 (+ 71us) service_pool.cc:225] Handling call
1026 02:56:51.313759 (+2436412us) ts_tablet_manager.cc:1151] Deleting tablet abf58185766448d1a7c4f3d304b0c5b1
1026 02:56:51.313767 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.313846 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2436235}
I20251026 02:56:51.314327 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1149) took 2436 ms. Trace:
I20251026 02:56:51.314393 2377 rpcz_store.cc:276] 1026 02:56:48.877396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.877452 (+ 56us) service_pool.cc:225] Handling call
1026 02:56:51.314217 (+2436765us) ts_tablet_manager.cc:1151] Deleting tablet b2eab97dc2b9409697589a34e9cc7d6c
1026 02:56:51.314227 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.314310 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2436583}
I20251026 02:56:51.314771 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1151) took 2437 ms. Trace:
I20251026 02:56:51.314846 2377 rpcz_store.cc:276] 1026 02:56:48.877606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.877643 (+ 37us) service_pool.cc:225] Handling call
1026 02:56:51.314681 (+2437038us) ts_tablet_manager.cc:1151] Deleting tablet 19016e8e11cc42c69ce44a819db9e298
1026 02:56:51.314689 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.314757 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2436890}
I20251026 02:56:51.315886 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1150) took 2438 ms. Trace:
I20251026 02:56:51.315961 2377 rpcz_store.cc:276] 1026 02:56:48.877499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.877685 (+ 186us) service_pool.cc:225] Handling call
1026 02:56:51.315788 (+2438103us) ts_tablet_manager.cc:1151] Deleting tablet 163b8f785675474e90ff90a1fe6fc056
1026 02:56:51.315796 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.315869 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2437960}
I20251026 02:56:51.316392 2376 raft_consensus.cc:2243] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:51.316565 2376 raft_consensus.cc:2272] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:51.317078 2376 ts_tablet_manager.cc:1916] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:51.317963 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 9d19dbc1fde04729be492963853de6bf (DEFAULT_TABLE table=default.Table_cbbe03e029b7424d99157ea61f5a01ce [id=9842bd55d8b140f9907797c2bb295d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.318634 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d19dbc1fde04729be492963853de6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.319003 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f4c327caa644d89098496bc324b976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:51.319929 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66ede86c604e47dbb982e5a3ba18bcb7" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.320042 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66ede86c604e47dbb982e5a3ba18bcb7" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:51.320501 1943 tablet_service.cc:1505] Processing CreateTablet for tablet 6bc01d5ee1284036b3c70d07c5e763e7 (DEFAULT_TABLE table=default.Table_272aa94f33a74e0f99feb2cb36453493 [id=b30d6e7092304ad5986e6a19cbbeae37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.320900 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet ceb7249968ff44c1b5be1d1d18605ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:51.321756 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 2e3df078612940cf86dd1da0eae248d9 (DEFAULT_TABLE table=default.Table_1ee28dad1e4148a8b092d79b5e5b2709 [id=6f7c232fc8ea49328f3e2fc3b46694ab]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:51.321946 2053 leader_election.cc:343] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.322058 2053 leader_election.cc:304] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:51.322106 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e3df078612940cf86dd1da0eae248d9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:51.322409 2053 leader_election.cc:343] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.322491 2053 leader_election.cc:304] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:51.322955 4098 raft_consensus.cc:2749] T c1dd78f088f84a509c761c934c039016 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:51.323107 4098 raft_consensus.cc:2749] T 0e05eada8742471f87d73f5d960817c0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:51.324474 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "027dd92fc5cf4c9e847de3e55e0dac1c" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.328835 1945 tablet_service.cc:1505] Processing CreateTablet for tablet d1ace19174ec4cca8314816665583cb2 (DEFAULT_TABLE table=default.Table_d744c8c07d3c4f25b019bfe49bf82b68 [id=1797a88a396d4209ba823205e75fc294]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.329485 2381 tablet_bootstrap.cc:492] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:51.329620 2381 ts_tablet_manager.cc:1403] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.098s user 0.005s sys 0.005s
I20251026 02:56:51.329918 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fa6400ae20a4b0d8a9a2013b3040ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.330288 2381 raft_consensus.cc:359] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.330456 2381 raft_consensus.cc:385] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:51.330616 2381 raft_consensus.cc:740] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:51.330840 2381 consensus_queue.cc:260] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.331436 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1ace19174ec4cca8314816665583cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.332336 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bc01d5ee1284036b3c70d07c5e763e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.332425 2381 ts_tablet_manager.cc:1434] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 02:56:51.333477 2381 tablet_bootstrap.cc:492] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:51.335039 2381 tablet_bootstrap.cc:654] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:51.340878 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a7b81b4ed2c49bfa1674d5775110545. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.342756 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 790a6b5db36c4fb2a83e8a6a43e5414b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.343761 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 45fd45ca857e45a097e86be7539f069c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:51.343637 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 5a00bc04b3654f90af342986751344e6 (table default.Table_164cb1784f4645e499339f7163fdfc2b [id=91b4b1a8162441299c680e7c202a96d9]) successfully deleted
I20251026 02:56:51.344785 2376 ts_tablet_manager.cc:1929] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:51.344900 2376 log.cc:1199] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/a012d868a78f4324a92db83c81931b9d
I20251026 02:56:51.345273 2376 ts_tablet_manager.cc:1950] T a012d868a78f4324a92db83c81931b9d P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:51.345590 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1030) took 3986 ms. Trace:
I20251026 02:56:51.345644 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b399723adb5d4ac0a842ebed6c2e029c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.345692 2376 rpcz_store.cc:276] 1026 02:56:47.359464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.361020 (+ 1556us) service_pool.cc:225] Handling call
1026 02:56:51.287537 (+3926517us) ts_tablet_manager.cc:1151] Deleting tablet a012d868a78f4324a92db83c81931b9d
1026 02:56:51.287547 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.333052 (+ 45505us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.344704 (+ 11652us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.345575 (+ 871us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28579,"tablet-delete.queue_time_us":3884690}
W20251026 02:56:51.345806 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet c0e6905e5ac147e3bc3a0073816faf76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0e6905e5ac147e3bc3a0073816faf76 already in progress: creating tablet
I20251026 02:56:51.346000 2376 tablet_replica.cc:333] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
W20251026 02:56:51.346002 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 8869062eedda4bfe99b1793ae0353680 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8869062eedda4bfe99b1793ae0353680 already in progress: creating tablet
W20251026 02:56:51.346179 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 4ed205c1800243ed9830d720ac85de2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ed205c1800243ed9830d720ac85de2f already in progress: creating tablet
I20251026 02:56:51.346231 2376 raft_consensus.cc:2243] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:51.346398 2376 raft_consensus.cc:2272] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:51.346967 2376 ts_tablet_manager.cc:1916] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:51.348951 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet abf58185766448d1a7c4f3d304b0c5b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abf58185766448d1a7c4f3d304b0c5b1 already in progress: creating tablet
W20251026 02:56:51.349182 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet b2eab97dc2b9409697589a34e9cc7d6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2eab97dc2b9409697589a34e9cc7d6c already in progress: creating tablet
W20251026 02:56:51.349365 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 19016e8e11cc42c69ce44a819db9e298 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19016e8e11cc42c69ce44a819db9e298 already in progress: creating tablet
W20251026 02:56:51.349535 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 163b8f785675474e90ff90a1fe6fc056 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 163b8f785675474e90ff90a1fe6fc056 already in progress: creating tablet
I20251026 02:56:51.350358 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 76c779bae091430fac46ad5dffb360ca (DEFAULT_TABLE table=default.Table_8e0fc984298540898b1ae289f95d504f [id=90b00dc4d553430c8b63dccd1a81ebd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.350795 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c779bae091430fac46ad5dffb360ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.352384 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 36fca225e913492e84a96043fa7ed215 (DEFAULT_TABLE table=default.Table_67618fda8a6d4b7a85112555b0961143 [id=cdfb39a67e904cef9266733ac9848c0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.352839 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fca225e913492e84a96043fa7ed215. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.353394 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 2e3df078612940cf86dd1da0eae248d9 (DEFAULT_TABLE table=default.Table_1ee28dad1e4148a8b092d79b5e5b2709 [id=6f7c232fc8ea49328f3e2fc3b46694ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.353690 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e3df078612940cf86dd1da0eae248d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.353647 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 01338cc196084f38b31bf5c41751ece1 (DEFAULT_TABLE table=default.Table_c3b49514cb69429ca8b6cc1fd3422ddf [id=a966e7a92b454d419fb12dab06067b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.353955 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01338cc196084f38b31bf5c41751ece1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.356527 2229 tablet_service.cc:1505] Processing CreateTablet for tablet ffe8d17204cb40a09992b5600958e64e (DEFAULT_TABLE table=default.Table_55b76386af9a4934813a5c92774f5697 [id=89d5f588892a42498df49590ec6e2f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.356701 2232 tablet_service.cc:1505] Processing CreateTablet for tablet ee46aae2e3304a3ea7ac5e02ea4b3b4b (DEFAULT_TABLE table=default.Table_9fef71477b964049a34fada596af80bf [id=240bb6da01cf4c1b98f9301e17cc7817]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.356877 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe8d17204cb40a09992b5600958e64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.357187 2233 tablet_service.cc:1505] Processing CreateTablet for tablet 047bfb3525054c44aeb7f37625d9d8b8 (DEFAULT_TABLE table=default.Table_7e72edb3045a47d1ab44c37eadf3c705 [id=140fa200b8934fd1b0f8098b27dd9a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.357553 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 047bfb3525054c44aeb7f37625d9d8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.359252 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet f719c720cc474181a63786a3e248ab2c (table default.Table_e586d81375544236ac7d95297fca01a8 [id=67731996dfc04572b0dfebf0edb71942]) successfully deleted
W20251026 02:56:51.359557 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 0aabfe11dedd454f8221fb5de25a268b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0aabfe11dedd454f8221fb5de25a268b already in progress: creating tablet
I20251026 02:56:51.360129 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_5240cf82ceb146fb9fc20241d2cb5761"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:51.360625 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 53e4ac96055a4de89485f3eeb5e1fcf2 (DEFAULT_TABLE table=default.Table_a09f60f754244ce3958f5633cf37ec42 [id=7036cf643bdf4e39a541d3c7ec37b53c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:51.360702 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5240cf82ceb146fb9fc20241d2cb5761 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.360950 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53e4ac96055a4de89485f3eeb5e1fcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.364604 4067 raft_consensus.cc:493] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.364764 4067 raft_consensus.cc:515] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.365326 4067 leader_election.cc:290] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.356536 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 9d19dbc1fde04729be492963853de6bf (DEFAULT_TABLE table=default.Table_cbbe03e029b7424d99157ea61f5a01ce [id=9842bd55d8b140f9907797c2bb295d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.365867 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet 142043b5d558490c9399626c3764ff7e with delete_type TABLET_DATA_DELETED (replaced by tablet f2febe856ba04f549c66ea7d95f54482) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:51.366170 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d19dbc1fde04729be492963853de6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.368366 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee46aae2e3304a3ea7ac5e02ea4b3b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.368517 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1072) took 4799 ms. Trace:
I20251026 02:56:51.368597 2372 rpcz_store.cc:276] 1026 02:56:46.568688 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.574938 (+ 6250us) service_pool.cc:225] Handling call
1026 02:56:51.368346 (+4793408us) ts_tablet_manager.cc:1151] Deleting tablet 1406c22ee4ee4f94acecad9546208e79
1026 02:56:51.368359 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.368474 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4792451}
W20251026 02:56:51.370029 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 1406c22ee4ee4f94acecad9546208e79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1406c22ee4ee4f94acecad9546208e79 already in progress: creating tablet
W20251026 02:56:51.373024 2195 leader_election.cc:343] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 868dbbfce40c4dd6b36cc220966b1986
I20251026 02:56:51.373392 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet a1d8bfca6a554e2abbbb9a0c7e2c0c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:51.376497 2375 consensus_queue.cc:260] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.381158 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 249c05690d8f4811904e455e1d078cf6
I20251026 02:56:51.381292 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet eff2a8d01a004fc89ba3421e10da6ff5
W20251026 02:56:51.381880 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b3bb056c94149bbb9f3dbfeb6db0d5a consensus state has no leader
W20251026 02:56:51.381158 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d7261ee6f244e77891e37a4c1a984ed consensus state has no leader
I20251026 02:56:51.387388 2375 ts_tablet_manager.cc:1434] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.154s user 0.002s sys 0.001s
I20251026 02:56:51.387648 1981 raft_consensus.cc:3060] T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.388662 2375 tablet_bootstrap.cc:492] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:51.397435 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet eff2a8d01a004fc89ba3421e10da6ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:51.397657 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet 249c05690d8f4811904e455e1d078cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:51.397511 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_cb422e5dd84d4d50ac55af4385a3b06b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.398270 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb422e5dd84d4d50ac55af4385a3b06b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.398556 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eabfc95caa84afc9a8956a7ac13fea6" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:51.399897 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ea364c4a5cf4f3f99d9ff6904add697" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:51.400076 2260 raft_consensus.cc:2468] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:51.401439 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b9a5daa36b547a7888d38dc3329eaf2" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:51.401607 2260 raft_consensus.cc:2468] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:51.402020 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1152) took 2524 ms. Trace:
I20251026 02:56:51.402102 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e9353957d5a046fc9f7a7e4316ff606b" } new_table_name: "default.Table_d14a01ea46f346e2aa335721052454fe" modify_external_catalogs: true
I20251026 02:56:51.402100 2377 rpcz_store.cc:276] 1026 02:56:48.877718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.877855 (+ 137us) service_pool.cc:225] Handling call
1026 02:56:51.316304 (+2438449us) ts_tablet_manager.cc:1151] Deleting tablet 551cc0916a21463d9190d14a31e79988
1026 02:56:51.316315 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.402007 (+ 85692us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2438314}
I20251026 02:56:51.402560 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1156) took 2456 ms. Trace:
I20251026 02:56:51.402628 2377 rpcz_store.cc:276] 1026 02:56:48.946422 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.948737 (+ 2315us) service_pool.cc:225] Handling call
1026 02:56:51.402449 (+2453712us) ts_tablet_manager.cc:1151] Deleting tablet fe27c85771964e4ca0ea747bbb4826df
1026 02:56:51.402459 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.402547 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2453508}
I20251026 02:56:51.402999 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1157) took 2424 ms. Trace:
I20251026 02:56:51.403039 1655 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 41a1442fffe54add93666e3c78a0cc58 (table default.Table_1faa0c8151724c5e816546ee2e1d5182 [id=a106f9ece0b9467a8bc1d5d4b3524c41]). Expected version 1 got 0
I20251026 02:56:51.403064 2377 rpcz_store.cc:276] 1026 02:56:48.978398 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.978463 (+ 65us) service_pool.cc:225] Handling call
1026 02:56:51.402914 (+2424451us) ts_tablet_manager.cc:1151] Deleting tablet cedff3287c7b4a07a81462e8c4b44970
1026 02:56:51.402922 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.402987 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2424242}
I20251026 02:56:51.403447 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1158) took 2424 ms. Trace:
I20251026 02:56:51.403605 2377 rpcz_store.cc:276] 1026 02:56:48.978524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.978722 (+ 198us) service_pool.cc:225] Handling call
1026 02:56:51.403332 (+2424610us) ts_tablet_manager.cc:1151] Deleting tablet 12e10fd463cd43c7abb22ef428c0aa37
1026 02:56:51.403340 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.403434 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2424429}
I20251026 02:56:51.403795 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_e88975df66e940a89e6353445cc36c7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:51.403981 2377 tablet_replica.cc:333] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:51.404165 2377 raft_consensus.cc:2243] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
W20251026 02:56:51.404301 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e88975df66e940a89e6353445cc36c7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.404500 2377 raft_consensus.cc:2272] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:51.405026 2377 ts_tablet_manager.cc:1916] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:51.405630 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 551cc0916a21463d9190d14a31e79988 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 551cc0916a21463d9190d14a31e79988 already in progress: creating tablet
W20251026 02:56:51.405818 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet fe27c85771964e4ca0ea747bbb4826df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe27c85771964e4ca0ea747bbb4826df already in progress: creating tablet
W20251026 02:56:51.405973 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet cedff3287c7b4a07a81462e8c4b44970 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cedff3287c7b4a07a81462e8c4b44970 already in progress: creating tablet
W20251026 02:56:51.406141 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 12e10fd463cd43c7abb22ef428c0aa37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e10fd463cd43c7abb22ef428c0aa37 already in progress: creating tablet
I20251026 02:56:51.406666 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2398d6b14bee449e97556f1c169d3f77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:51.407169 2081 tablet_service.cc:1552] Processing DeleteTablet for tablet 249c05690d8f4811904e455e1d078cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:51.407205 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2398d6b14bee449e97556f1c169d3f77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.407423 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet eff2a8d01a004fc89ba3421e10da6ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:51.407745 2372 tablet_replica.cc:333] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:51.407980 2372 raft_consensus.cc:2243] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:51.408165 2372 raft_consensus.cc:2272] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:51.408852 2372 ts_tablet_manager.cc:1916] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:51.413944 4302 raft_consensus.cc:493] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b6b0a6c4e144982b67980118cddba48)
I20251026 02:56:51.414095 4302 raft_consensus.cc:515] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.414710 4302 leader_election.cc:290] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.416908 4302 raft_consensus.cc:493] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.417050 4302 raft_consensus.cc:515] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.418277 1915 leader_election.cc:304] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:51.418535 4305 raft_consensus.cc:2749] T 1eabfc95caa84afc9a8956a7ac13fea6 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251026 02:56:51.418949 1915 consensus_peers.cc:597] T 9a345fa5edbd48d2891b8b40b09cb3f8 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Couldn't send request to peer 696476265495492790739ec126a08539. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a345fa5edbd48d2891b8b40b09cb3f8. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:51.419215 1915 leader_election.cc:304] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:51.419493 4305 raft_consensus.cc:2804] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:51.419572 4305 raft_consensus.cc:493] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:51.419625 4305 raft_consensus.cc:3060] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.420043 1915 leader_election.cc:304] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:51.420332 4374 raft_consensus.cc:2804] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:51.420396 4374 raft_consensus.cc:493] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:51.420449 4374 raft_consensus.cc:3060] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.421177 4376 consensus_queue.cc:1048] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:51.424783 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f21dfb250b814ac28bce8fad0ec9d7bb has no consensus state
I20251026 02:56:51.424961 2378 tablet_bootstrap.cc:654] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:51.425758 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aed4584b49a4eccac2ba44454866d49" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.426632 4302 leader_election.cc:290] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.427318 4374 raft_consensus.cc:515] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.428001 4374 leader_election.cc:290] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.432374 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet a012d868a78f4324a92db83c81931b9d (table default.Table_3dd70def7f1046ea9f2b5bbb7a74e4cb [id=b4e3ea5575a140ac85b4c0cbd57c9ec4]) successfully deleted
W20251026 02:56:51.437276 1916 leader_election.cc:343] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 1fb2ff87cf0248a0a4a77f2733ef867b
I20251026 02:56:51.437476 2376 ts_tablet_manager.cc:1929] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:51.437595 2376 log.cc:1199] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/b2bfd9b55c484ef8a9c6345d56f794fb
I20251026 02:56:51.437918 2376 ts_tablet_manager.cc:1950] T b2bfd9b55c484ef8a9c6345d56f794fb P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
W20251026 02:56:51.438740 1916 leader_election.cc:343] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 9b9a5daa36b547a7888d38dc3329eaf2
I20251026 02:56:51.438791 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b77bb6b4722449c9699b548a9ef6314" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:51.439656 3997 raft_consensus.cc:493] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.439790 3997 raft_consensus.cc:515] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.440449 3997 leader_election.cc:290] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:51.440770 1915 leader_election.cc:343] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 1fb2ff87cf0248a0a4a77f2733ef867b
I20251026 02:56:51.443522 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1034) took 3983 ms. Trace:
I20251026 02:56:51.443620 2376 rpcz_store.cc:276] 1026 02:56:47.460297 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.460351 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:51.345957 (+3885606us) ts_tablet_manager.cc:1151] Deleting tablet b2bfd9b55c484ef8a9c6345d56f794fb
1026 02:56:51.345965 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.431646 (+ 85681us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.437371 (+ 5725us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.443502 (+ 6131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3885417}
I20251026 02:56:51.443986 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1035) took 3963 ms. Trace:
I20251026 02:56:51.444053 2376 rpcz_store.cc:276] 1026 02:56:47.480383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.480457 (+ 74us) service_pool.cc:225] Handling call
1026 02:56:51.443875 (+3963418us) ts_tablet_manager.cc:1151] Deleting tablet 97aa7f0266854c0a8f17e6bceac0c9b5
1026 02:56:51.443883 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.443974 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3963195}
I20251026 02:56:51.444383 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1036) took 3947 ms. Trace:
I20251026 02:56:51.444469 2376 rpcz_store.cc:276] 1026 02:56:47.497351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.497404 (+ 53us) service_pool.cc:225] Handling call
1026 02:56:51.444281 (+3946877us) ts_tablet_manager.cc:1151] Deleting tablet f840050f7e314d74ac6e604354a7d4c9
1026 02:56:51.444292 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.444366 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3933019}
I20251026 02:56:51.444736 2376 tablet_replica.cc:333] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:51.444967 2376 raft_consensus.cc:2243] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:51.445214 2376 raft_consensus.cc:2272] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:51.445910 2376 ts_tablet_manager.cc:1916] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:51.448091 1942 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58886 (request call id 188) took 1286 ms. Trace:
I20251026 02:56:51.448176 1942 rpcz_store.cc:276] 1026 02:56:50.161496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:50.228937 (+ 67441us) service_pool.cc:225] Handling call
1026 02:56:50.487841 (+258904us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x6160000f8254. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:50.647529 (+159688us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x6160000f2e54. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:51.448070 (+800541us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":293011072}
W20251026 02:56:51.448390 2195 consensus_peers.cc:597] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 209fa71af13142a58efb9f3dab3eecd3. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:51.453898 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_85fc65eae0ab485abad03396cbe30fef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:51.454288 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3597ea2076244c4ba6e67599f245a639" } new_table_name: "default.Table_3c82c900dc6d4eabbf65a1e65baeb55b" modify_external_catalogs: true
W20251026 02:56:51.454514 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85fc65eae0ab485abad03396cbe30fef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.455540 4305 raft_consensus.cc:515] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.456401 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_dcb798e05db24976a7f83acce835a73a" } new_table_name: "default.Table_f0758cabc30346959fcaac8df0a440a0" modify_external_catalogs: true
I20251026 02:56:51.456569 3997 raft_consensus.cc:493] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.456745 3997 raft_consensus.cc:515] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.456905 1655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_338870d29792464aa2ab8c8c2f5978ac" } modify_external_catalogs: true
I20251026 02:56:51.457031 1655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_338870d29792464aa2ab8c8c2f5978ac" } modify_external_catalogs: true
I20251026 02:56:51.457406 3997 leader_election.cc:290] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.458284 1915 leader_election.cc:304] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:51.458431 4305 leader_election.cc:290] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.458559 4305 raft_consensus.cc:2749] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251026 02:56:51.459579 1916 leader_election.cc:343] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 6ea364c4a5cf4f3f99d9ff6904add697
I20251026 02:56:51.462908 2376 ts_tablet_manager.cc:1929] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:51.463150 1945 tablet_service.cc:1505] Processing CreateTablet for tablet ffe8d17204cb40a09992b5600958e64e (DEFAULT_TABLE table=default.Table_55b76386af9a4934813a5c92774f5697 [id=89d5f588892a42498df49590ec6e2f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.463459 2376 log.cc:1199] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/ce1ab21ddf8e45bc9063b63db9c1b97f
I20251026 02:56:51.465826 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 36fca225e913492e84a96043fa7ed215 (DEFAULT_TABLE table=default.Table_67618fda8a6d4b7a85112555b0961143 [id=cdfb39a67e904cef9266733ac9848c0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.466637 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe8d17204cb40a09992b5600958e64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.467361 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_856686a17ddb4390908247e68a52742e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.467885 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_856686a17ddb4390908247e68a52742e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.469549 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fca225e913492e84a96043fa7ed215. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.470137 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b9a5daa36b547a7888d38dc3329eaf2" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:51.470242 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b77bb6b4722449c9699b548a9ef6314" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:51.470321 2262 raft_consensus.cc:3060] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.470618 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ea364c4a5cf4f3f99d9ff6904add697" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:51.470742 2256 raft_consensus.cc:3060] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.471697 2376 ts_tablet_manager.cc:1950] T ce1ab21ddf8e45bc9063b63db9c1b97f P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:51.471653 1915 leader_election.cc:304] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:51.471992 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1041) took 3753 ms. Trace:
I20251026 02:56:51.472064 2376 rpcz_store.cc:276] 1026 02:56:47.718872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.721260 (+ 2388us) service_pool.cc:225] Handling call
1026 02:56:51.444685 (+3723425us) ts_tablet_manager.cc:1151] Deleting tablet ce1ab21ddf8e45bc9063b63db9c1b97f
1026 02:56:51.444695 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.450807 (+ 6112us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.462791 (+ 11984us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.471975 (+ 9184us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3723205}
I20251026 02:56:51.472345 4378 raft_consensus.cc:2749] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:51.472450 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1068) took 3290 ms. Trace:
I20251026 02:56:51.472525 2376 rpcz_store.cc:276] 1026 02:56:48.182302 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.182349 (+ 47us) service_pool.cc:225] Handling call
1026 02:56:51.472342 (+3289993us) ts_tablet_manager.cc:1151] Deleting tablet c0e6905e5ac147e3bc3a0073816faf76
1026 02:56:51.472351 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.472435 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3275954}
I20251026 02:56:51.472774 2376 tablet_replica.cc:333] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:51.472935 2376 raft_consensus.cc:2243] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:51.473075 2376 raft_consensus.cc:2272] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:51.473487 2256 raft_consensus.cc:2468] T 6ea364c4a5cf4f3f99d9ff6904add697 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:51.473610 2376 ts_tablet_manager.cc:1916] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:51.473636 2262 raft_consensus.cc:2468] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:51.473946 1915 leader_election.cc:304] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:51.474200 4378 raft_consensus.cc:2804] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:51.474299 4378 raft_consensus.cc:697] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:51.474536 4378 consensus_queue.cc:237] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.474676 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2dbe045251c840efbf656f41a60ad6aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:51.475152 1915 leader_election.cc:304] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
W20251026 02:56:51.475232 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2dbe045251c840efbf656f41a60ad6aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.475346 4305 raft_consensus.cc:2804] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:51.475445 4305 raft_consensus.cc:697] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:51.475657 4305 consensus_queue.cc:237] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.477779 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ca4a98dc25144099a2ca90bb12766232" } modify_external_catalogs: true
I20251026 02:56:51.477962 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_ca4a98dc25144099a2ca90bb12766232" } modify_external_catalogs: true
I20251026 02:56:51.477797 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fce269332a224a298ef59f900f8c8d03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.479516 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fce269332a224a298ef59f900f8c8d03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.480739 2377 ts_tablet_manager.cc:1929] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:51.482868 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cbbe03e029b7424d99157ea61f5a01ce" } modify_external_catalogs: true
I20251026 02:56:51.483021 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_cbbe03e029b7424d99157ea61f5a01ce" } modify_external_catalogs: true
I20251026 02:56:51.488955 2110 raft_consensus.cc:1275] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Refusing update from remote peer 696476265495492790739ec126a08539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 02:56:51.489617 4069 consensus_queue.cc:1048] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:51.494881 1655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a8749da375204fe0b3dcc95345ec995e
W20251026 02:56:51.495630 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8af5545da3e4241a588465b1783a8eb consensus state has no leader
I20251026 02:56:51.495811 2084 tablet_service.cc:1552] Processing DeleteTablet for tablet a8749da375204fe0b3dcc95345ec995e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:51.495940 1663 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet eb3d0388b264447ea887452856934420
W20251026 02:56:51.496574 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb40f45aebb5463b99ab53590cc10185 has no consensus state
I20251026 02:56:51.498342 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet a8749da375204fe0b3dcc95345ec995e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:51.498585 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet a8749da375204fe0b3dcc95345ec995e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:51.499785 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet b2bfd9b55c484ef8a9c6345d56f794fb (table default.Table_30e98b21e65d443097e2fc01f96f4780 [id=0d9be16d1aec415a8f9a2433f8d7ac6b]) successfully deleted
W20251026 02:56:51.499986 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 97aa7f0266854c0a8f17e6bceac0c9b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97aa7f0266854c0a8f17e6bceac0c9b5 already in progress: creating tablet
W20251026 02:56:51.500128 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet f840050f7e314d74ac6e604354a7d4c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f840050f7e314d74ac6e604354a7d4c9 already in progress: creating tablet
I20251026 02:56:51.504455 2376 ts_tablet_manager.cc:1929] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:51.504577 2376 log.cc:1199] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/5a00bc04b3654f90af342986751344e6
I20251026 02:56:51.505797 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet ce1ab21ddf8e45bc9063b63db9c1b97f (table default.Table_7c080afce3474539b73dd8c734adef21 [id=2e75e06786d940d092bdc81772be0dd5]) successfully deleted
W20251026 02:56:51.506053 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet c0e6905e5ac147e3bc3a0073816faf76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0e6905e5ac147e3bc3a0073816faf76 already in progress: creating tablet
I20251026 02:56:51.507740 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 01338cc196084f38b31bf5c41751ece1 (DEFAULT_TABLE table=default.Table_c3b49514cb69429ca8b6cc1fd3422ddf [id=a966e7a92b454d419fb12dab06067b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.508129 2376 ts_tablet_manager.cc:1950] T 5a00bc04b3654f90af342986751344e6 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:51.508919 2377 log.cc:1199] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/4b5e61afe90e4bc4a2897de21e0b3919
I20251026 02:56:51.509433 2377 ts_tablet_manager.cc:1950] T 4b5e61afe90e4bc4a2897de21e0b3919 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:51.511301 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1160) took 2532 ms. Trace:
I20251026 02:56:51.516602 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01338cc196084f38b31bf5c41751ece1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:51.542608 1916 consensus_peers.cc:597] T e89a98fcfdbb4011876b0ebc2af91f67 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e89a98fcfdbb4011876b0ebc2af91f67. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:51.544045 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 2e3df078612940cf86dd1da0eae248d9 (DEFAULT_TABLE table=default.Table_1ee28dad1e4148a8b092d79b5e5b2709 [id=6f7c232fc8ea49328f3e2fc3b46694ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.544623 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e3df078612940cf86dd1da0eae248d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.544482 2377 rpcz_store.cc:276] 1026 02:56:48.978686 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.978932 (+ 246us) service_pool.cc:225] Handling call
1026 02:56:51.403939 (+2425007us) ts_tablet_manager.cc:1151] Deleting tablet 4b5e61afe90e4bc4a2897de21e0b3919
1026 02:56:51.403947 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.449782 (+ 45835us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.480626 (+ 30844us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.511275 (+ 30649us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2424886}
I20251026 02:56:51.544955 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 4b5e61afe90e4bc4a2897de21e0b3919 (table default.Table_ca64283c7621400ba743ec5cff194692 [id=9fd6e480fc3543149959773e691a3ddd]) successfully deleted
I20251026 02:56:51.545018 2377 tablet_replica.cc:333] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:51.545405 2377 raft_consensus.cc:2243] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:51.545639 2377 raft_consensus.cc:2272] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:51.546202 2377 ts_tablet_manager.cc:1916] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:51.551801 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet eff2a8d01a004fc89ba3421e10da6ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:51.555001 1916 consensus_peers.cc:597] T 1dbbfb77851d443f8cf4d798f3e07e59 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dbbfb77851d443f8cf4d798f3e07e59. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:51.561674 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 047bfb3525054c44aeb7f37625d9d8b8 (DEFAULT_TABLE table=default.Table_7e72edb3045a47d1ab44c37eadf3c705 [id=140fa200b8934fd1b0f8098b27dd9a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.562309 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 047bfb3525054c44aeb7f37625d9d8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.567499 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 53e4ac96055a4de89485f3eeb5e1fcf2 (DEFAULT_TABLE table=default.Table_a09f60f754244ce3958f5633cf37ec42 [id=7036cf643bdf4e39a541d3c7ec37b53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.567951 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53e4ac96055a4de89485f3eeb5e1fcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.569121 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 9d19dbc1fde04729be492963853de6bf (DEFAULT_TABLE table=default.Table_cbbe03e029b7424d99157ea61f5a01ce [id=9842bd55d8b140f9907797c2bb295d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.569589 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d19dbc1fde04729be492963853de6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.579411 1943 tablet_service.cc:1552] Processing DeleteTablet for tablet 249c05690d8f4811904e455e1d078cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:51.581537 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1067) took 3399 ms. Trace:
I20251026 02:56:51.581641 2376 rpcz_store.cc:276] 1026 02:56:48.182155 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.190517 (+ 8362us) service_pool.cc:225] Handling call
1026 02:56:51.472736 (+3282219us) ts_tablet_manager.cc:1151] Deleting tablet 5a00bc04b3654f90af342986751344e6
1026 02:56:51.472744 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.489936 (+ 17192us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.504335 (+ 14399us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.581422 (+ 77087us) spinlock_profiling.cc:243] Waited 7.82 ms on lock 0x61800000d518. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc41a1782fd 00007fc41a17936e 00007fc42ddabc9d 00007fc42dd2a203 00007fc429f5f6f3 00007fc429f922d4 00007fc429f72d70 00007fc429f72afd 0000000000a8867c 00007fc41a1d5d7f
1026 02:56:51.581514 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17976704,"tablet-delete.queue_time_us":3269746}
I20251026 02:56:51.582080 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1069) took 3399 ms. Trace:
I20251026 02:56:51.582166 2376 rpcz_store.cc:276] 1026 02:56:48.182901 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.182957 (+ 56us) service_pool.cc:225] Handling call
1026 02:56:51.581963 (+3399006us) ts_tablet_manager.cc:1151] Deleting tablet 086b40036f0c4c97bb95647f91ce7939
1026 02:56:51.581972 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.582064 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3378925}
I20251026 02:56:51.582464 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1066) took 3400 ms. Trace:
I20251026 02:56:51.582545 2376 rpcz_store.cc:276] 1026 02:56:48.181898 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.184463 (+ 2565us) service_pool.cc:225] Handling call
1026 02:56:51.582373 (+3397910us) ts_tablet_manager.cc:1151] Deleting tablet 599a88ec0e76409c97ab62d6f55ebcd0
1026 02:56:51.582381 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.582450 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3369319}
I20251026 02:56:51.582890 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1070) took 3302 ms. Trace:
I20251026 02:56:51.582973 2376 rpcz_store.cc:276] 1026 02:56:48.280722 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.281278 (+ 556us) service_pool.cc:225] Handling call
1026 02:56:51.582790 (+3301512us) ts_tablet_manager.cc:1151] Deleting tablet 8869062eedda4bfe99b1793ae0353680
1026 02:56:51.582800 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.582875 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3301265}
I20251026 02:56:51.583278 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1071) took 3280 ms. Trace:
I20251026 02:56:51.583357 2376 rpcz_store.cc:276] 1026 02:56:48.302724 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.303829 (+ 1105us) service_pool.cc:225] Handling call
1026 02:56:51.583181 (+3279352us) ts_tablet_manager.cc:1151] Deleting tablet 4ed205c1800243ed9830d720ac85de2f
1026 02:56:51.583189 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.583265 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3279056}
I20251026 02:56:51.583554 2376 tablet_replica.cc:333] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:51.583711 2376 raft_consensus.cc:2243] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:51.583974 2376 raft_consensus.cc:2272] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:51.584867 2237 tablet_service.cc:1505] Processing CreateTablet for tablet be3ac522207d41e6915e5b9a580c6005 (DEFAULT_TABLE table=default.Table_35d854aa640a4cddbbbada2647659531 [id=ae56425a2f474112a855c01589332042]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.585312 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be3ac522207d41e6915e5b9a580c6005. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.586815 2084 tablet_service.cc:1505] Processing CreateTablet for tablet be3ac522207d41e6915e5b9a580c6005 (DEFAULT_TABLE table=default.Table_35d854aa640a4cddbbbada2647659531 [id=ae56425a2f474112a855c01589332042]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.586925 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 5a00bc04b3654f90af342986751344e6 (table default.Table_164cb1784f4645e499339f7163fdfc2b [id=91b4b1a8162441299c680e7c202a96d9]) successfully deleted
W20251026 02:56:51.587240 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 599a88ec0e76409c97ab62d6f55ebcd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 599a88ec0e76409c97ab62d6f55ebcd0 already in progress: creating tablet
W20251026 02:56:51.587421 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 8869062eedda4bfe99b1793ae0353680 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8869062eedda4bfe99b1793ae0353680 already in progress: creating tablet
W20251026 02:56:51.587574 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 4ed205c1800243ed9830d720ac85de2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ed205c1800243ed9830d720ac85de2f already in progress: creating tablet
I20251026 02:56:51.588137 2081 tablet_service.cc:1505] Processing CreateTablet for tablet cfee26e6f5c246ae9da6d180432aafcd (DEFAULT_TABLE table=default.Table_031fab6bd5b5488fb13986995c2dd70c [id=2813417827c64631adbf1fb36eeda1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.588388 2229 tablet_service.cc:1505] Processing CreateTablet for tablet cfee26e6f5c246ae9da6d180432aafcd (DEFAULT_TABLE table=default.Table_031fab6bd5b5488fb13986995c2dd70c [id=2813417827c64631adbf1fb36eeda1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.588737 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfee26e6f5c246ae9da6d180432aafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.589726 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 9d19dbc1fde04729be492963853de6bf
I20251026 02:56:51.590577 2082 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d19dbc1fde04729be492963853de6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:51.592152 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 2ac6e19b04974fd585822340c0bf6169 (DEFAULT_TABLE table=default.Table_d89900bf7bf04855a2bd970460c420e8 [id=fabca86bb32941c7883bd8e9e6a239aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.592339 2376 ts_tablet_manager.cc:1916] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:51.592531 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 2ac6e19b04974fd585822340c0bf6169 (DEFAULT_TABLE table=default.Table_d89900bf7bf04855a2bd970460c420e8 [id=fabca86bb32941c7883bd8e9e6a239aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.592854 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ac6e19b04974fd585822340c0bf6169. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.593612 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 04265a5178034dcaa2bb14b1efa3e493 (DEFAULT_TABLE table=default.Table_9f6ae0bb97e0495895985e078c5e79b0 [id=ee90f26d898241aab04a07df7a453f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.594017 4400 tablet.cc:1722] T 109073eb664d4090b7a2b786146e13fa P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251026 02:56:51.594727 4399 tablet.cc:1722] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251026 02:56:51.596432 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d19dbc1fde04729be492963853de6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:51.596496 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9304ae0aeabf48e99b2f10c5fba8cd1e" } modify_external_catalogs: true
I20251026 02:56:51.596674 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9304ae0aeabf48e99b2f10c5fba8cd1e" } modify_external_catalogs: true
I20251026 02:56:51.596829 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 04265a5178034dcaa2bb14b1efa3e493 (DEFAULT_TABLE table=default.Table_9f6ae0bb97e0495895985e078c5e79b0 [id=ee90f26d898241aab04a07df7a453f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.596907 1657 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_03d048a39e0548078a18eb96c4830780" } new_table_name: "default.Table_248d2af4d5994a42b62546bc2840f097" modify_external_catalogs: true
I20251026 02:56:51.597191 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04265a5178034dcaa2bb14b1efa3e493. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.597983 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_5fb9a6abaf3b49ba80d560ab9ca37142" } new_table_name: "default.Table_ac9afcac152340fbb486392806532471" modify_external_catalogs: true
I20251026 02:56:51.598308 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6c9225d34cdf4175bd75393736817b68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.598757 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c9225d34cdf4175bd75393736817b68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.599136 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_09fa02d45d344077a3a72842956fa545"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.599570 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09fa02d45d344077a3a72842956fa545 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.599825 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_eb26fd925b0748debe01dd229ed39094"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.600308 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb26fd925b0748debe01dd229ed39094 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.598031 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f2a821bbac24429d942cbb9a0d7e5b08" } modify_external_catalogs: true
I20251026 02:56:51.603897 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f2a821bbac24429d942cbb9a0d7e5b08" } modify_external_catalogs: true
I20251026 02:56:51.604300 2375 tablet_bootstrap.cc:654] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:51.605347 4305 raft_consensus.cc:493] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.605559 4305 raft_consensus.cc:515] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.606758 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc4ca4423e3a4e0c96846babec931fec" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:51.606977 2110 raft_consensus.cc:2393] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fef01f4f2b843538ea245a09dd087a8 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:51.608050 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc4ca4423e3a4e0c96846babec931fec" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:51.608579 1915 leader_election.cc:304] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:51.608844 4374 raft_consensus.cc:3060] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.609539 4305 leader_election.cc:290] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.610580 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7060cb495eb1423b9e3ecbd17c325948
I20251026 02:56:51.611505 2096 tablet_service.cc:1552] Processing DeleteTablet for tablet 7060cb495eb1423b9e3ecbd17c325948 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:51.611835 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4010e810fee5467db35ec065f4f0ae89 consensus state has no leader
I20251026 02:56:51.612628 2084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be3ac522207d41e6915e5b9a580c6005. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.612908 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ac6e19b04974fd585822340c0bf6169. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.613159 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04265a5178034dcaa2bb14b1efa3e493. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.620182 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 7060cb495eb1423b9e3ecbd17c325948 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:51.620805 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 76c779bae091430fac46ad5dffb360ca (DEFAULT_TABLE table=default.Table_8e0fc984298540898b1ae289f95d504f [id=90b00dc4d553430c8b63dccd1a81ebd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.621387 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c779bae091430fac46ad5dffb360ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.621908 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfee26e6f5c246ae9da6d180432aafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.624637 1955 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d19dbc1fde04729be492963853de6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:51.628151 1954 tablet_service.cc:1505] Processing CreateTablet for tablet cfee26e6f5c246ae9da6d180432aafcd (DEFAULT_TABLE table=default.Table_031fab6bd5b5488fb13986995c2dd70c [id=2813417827c64631adbf1fb36eeda1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.628656 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfee26e6f5c246ae9da6d180432aafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.630055 2232 tablet_service.cc:1505] Processing CreateTablet for tablet f21dfb250b814ac28bce8fad0ec9d7bb (DEFAULT_TABLE table=default.Table_d14a01ea46f346e2aa335721052454fe [id=567f5a6f3b6641e388d577fb768ca7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.630477 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f21dfb250b814ac28bce8fad0ec9d7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.633958 2094 tablet_service.cc:1505] Processing CreateTablet for tablet f21dfb250b814ac28bce8fad0ec9d7bb (DEFAULT_TABLE table=default.Table_d14a01ea46f346e2aa335721052454fe [id=567f5a6f3b6641e388d577fb768ca7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.634567 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f21dfb250b814ac28bce8fad0ec9d7bb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:51.638080 2194 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 316814us
W20251026 02:56:51.640642 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 317819us
I20251026 02:56:51.640599 4255 consensus_queue.cc:1048] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:51.641152 4255 raft_consensus.cc:493] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.641273 4255 raft_consensus.cc:515] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.641810 4255 leader_election.cc:290] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.642001 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00a8f0594509497a9cb856f7890da302" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:51.642060 4098 raft_consensus.cc:493] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.642200 4098 raft_consensus.cc:515] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.642342 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e830f1aa1270420c9b28a0491a1a0960" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:51.642932 4098 leader_election.cc:290] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.643218 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3af2695c41a4439bc4092b7cb1b2edf" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:51.643388 2109 raft_consensus.cc:2468] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:51.643862 2195 leader_election.cc:304] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:51.644079 4255 raft_consensus.cc:2804] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:51.644151 4255 raft_consensus.cc:493] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:51.644345 4255 raft_consensus.cc:3060] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.647143 4255 raft_consensus.cc:515] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.647610 2230 tablet_service.cc:1505] Processing CreateTablet for tablet b084395d643a4cb5b9a6f900d8a871bf (DEFAULT_TABLE table=default.Table_48fb5691d1674c79a437be2759a72637 [id=cad5aa4b07a948d4b5b720fd42468c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.647735 4255 leader_election.cc:290] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.648000 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b084395d643a4cb5b9a6f900d8a871bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.648571 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3af2695c41a4439bc4092b7cb1b2edf" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:51.648758 2109 raft_consensus.cc:3060] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.651415 2109 raft_consensus.cc:2468] T a3af2695c41a4439bc4092b7cb1b2edf P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:51.654058 2082 tablet_service.cc:1505] Processing CreateTablet for tablet b084395d643a4cb5b9a6f900d8a871bf (DEFAULT_TABLE table=default.Table_48fb5691d1674c79a437be2759a72637 [id=cad5aa4b07a948d4b5b720fd42468c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.654461 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b084395d643a4cb5b9a6f900d8a871bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.653497 2195 leader_election.cc:304] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:51.655566 4255 raft_consensus.cc:2804] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:51.655702 4255 raft_consensus.cc:697] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:51.655957 4255 consensus_queue.cc:237] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.657087 4098 raft_consensus.cc:493] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.657194 4098 raft_consensus.cc:515] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.657652 4098 leader_election.cc:290] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:51.658119 2053 leader_election.cc:343] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.658241 2053 leader_election.cc:304] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:51.658420 4098 raft_consensus.cc:2749] T 027dd92fc5cf4c9e847de3e55e0dac1c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:51.659117 2053 leader_election.cc:343] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.659232 2053 leader_election.cc:304] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:51.659520 4098 raft_consensus.cc:2749] T 3aed4584b49a4eccac2ba44454866d49 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:51.660426 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "142043b5d558490c9399626c3764ff7e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:51.660539 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55d8d698ef0f4a81a0e269943fa41206" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
W20251026 02:56:51.661072 2053 leader_election.cc:343] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:51.661239 2053 leader_election.cc:343] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:51.661790 2053 leader_election.cc:343] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:51.662056 2053 leader_election.cc:343] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.667971 1951 tablet_service.cc:1505] Processing CreateTablet for tablet ee46aae2e3304a3ea7ac5e02ea4b3b4b (DEFAULT_TABLE table=default.Table_9fef71477b964049a34fada596af80bf [id=240bb6da01cf4c1b98f9301e17cc7817]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.668620 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee46aae2e3304a3ea7ac5e02ea4b3b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.669668 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 56b4b784c9e2403e8a3c7d31859568c0 (DEFAULT_TABLE table=default.Table_9f662442aecd4fe5a7d273b0c1e6aca6 [id=0d0d8b1c451b43e5bd8876f697bebf0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.670053 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b4b784c9e2403e8a3c7d31859568c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.670123 2237 tablet_service.cc:1505] Processing CreateTablet for tablet 56b4b784c9e2403e8a3c7d31859568c0 (DEFAULT_TABLE table=default.Table_9f662442aecd4fe5a7d273b0c1e6aca6 [id=0d0d8b1c451b43e5bd8876f697bebf0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.670462 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b4b784c9e2403e8a3c7d31859568c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.672806 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 4010e810fee5467db35ec065f4f0ae89 (DEFAULT_TABLE table=default.Table_ac9afcac152340fbb486392806532471 [id=5086df93e533428185465ec6150122b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.672852 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 4010e810fee5467db35ec065f4f0ae89 (DEFAULT_TABLE table=default.Table_ac9afcac152340fbb486392806532471 [id=5086df93e533428185465ec6150122b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.673125 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4010e810fee5467db35ec065f4f0ae89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.673200 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4010e810fee5467db35ec065f4f0ae89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.677062 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 61a931473df24f97a75200450ac7f3b6 (DEFAULT_TABLE table=default.Table_d72028b7f087490dbd200e4172900e7e [id=4dba4794ab354a2e90c7102138335727]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.677448 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61a931473df24f97a75200450ac7f3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.684128 1943 tablet_service.cc:1505] Processing CreateTablet for tablet be3ac522207d41e6915e5b9a580c6005 (DEFAULT_TABLE table=default.Table_35d854aa640a4cddbbbada2647659531 [id=ae56425a2f474112a855c01589332042]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.687023 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 2c244e735f244909b91b0d0b41eee7b4 (DEFAULT_TABLE table=default.Table_e6e2339629ed45d39e9070c3e05c109d [id=584d27528e1747d3afc32789cf243bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.687837 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c244e735f244909b91b0d0b41eee7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.689589 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 2c244e735f244909b91b0d0b41eee7b4 (DEFAULT_TABLE table=default.Table_e6e2339629ed45d39e9070c3e05c109d [id=584d27528e1747d3afc32789cf243bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.690045 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c244e735f244909b91b0d0b41eee7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.693845 2377 ts_tablet_manager.cc:1929] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:51.693874 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 014f5a300c6a4cc88a2bf50fc85680af (DEFAULT_TABLE table=default.Table_94bf0678dbe746219b474992c5acde07 [id=b3dc2a9b971f4bd2851bec852e662fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.694401 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 014f5a300c6a4cc88a2bf50fc85680af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.695376 1658 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_20d4fe3113a8478bb9284318fbe94aeb" } new_table_name: "default.Table_b2886b13314242f79fd50c23ddf0c705" modify_external_catalogs: true
I20251026 02:56:51.696728 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0e87fae7410549a88fc45e95876f13ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.696938 1650 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet b8af5545da3e4241a588465b1783a8eb (table default.Table_3c82c900dc6d4eabbf65a1e65baeb55b [id=4ccfea55427149d98986456d6f5ef970], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b8af5545da3e4241a588465b1783a8eb consensus state has no leader
W20251026 02:56:51.697182 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e87fae7410549a88fc45e95876f13ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.701807 4067 raft_consensus.cc:493] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.701962 4067 raft_consensus.cc:515] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.702522 2375 tablet_bootstrap.cc:492] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:51.702530 4067 leader_election.cc:290] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.702682 2375 ts_tablet_manager.cc:1403] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.314s user 0.002s sys 0.001s
I20251026 02:56:51.703429 2375 raft_consensus.cc:359] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.703606 2375 raft_consensus.cc:385] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:51.703666 2375 raft_consensus.cc:740] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:51.703609 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_39cd69856c0a4284b604da1e447e172a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:51.703891 2375 consensus_queue.cc:260] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:51.704192 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39cd69856c0a4284b604da1e447e172a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.705461 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8fe7825fa8d46669772cf84bc4b3afc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:51.705420 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_fdaf051a8a4b4899a92c293e6428e200"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:51.705615 2109 raft_consensus.cc:2468] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:51.705894 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdaf051a8a4b4899a92c293e6428e200 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.706023 2195 leader_election.cc:304] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:51.706485 4067 raft_consensus.cc:2804] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:51.706555 4067 raft_consensus.cc:493] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:51.706609 4067 raft_consensus.cc:3060] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.696681 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_22fd677885264f808e07ce3b8ca2c5c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.708261 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22fd677885264f808e07ce3b8ca2c5c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.709237 4067 raft_consensus.cc:515] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.710438 4067 leader_election.cc:290] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.711681 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_814f1eb925e748a6a60c9e85eed9667f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.712365 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_814f1eb925e748a6a60c9e85eed9667f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.712747 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8fe7825fa8d46669772cf84bc4b3afc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:51.712939 2114 raft_consensus.cc:3060] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.713584 4098 raft_consensus.cc:493] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.713711 4098 raft_consensus.cc:515] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.714293 4098 leader_election.cc:290] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:51.715576 2194 leader_election.cc:343] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:51.715879 2194 leader_election.cc:343] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.716033 2114 raft_consensus.cc:2468] T b8fe7825fa8d46669772cf84bc4b3afc P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:51.721237 2195 leader_election.cc:304] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:51.722035 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_20aea86003f649d282631fee5df071c8" } modify_external_catalogs: true
I20251026 02:56:51.722220 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_20aea86003f649d282631fee5df071c8" } modify_external_catalogs: true
I20251026 02:56:51.724565 4255 raft_consensus.cc:2804] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:51.724699 4255 raft_consensus.cc:697] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:51.724989 4255 consensus_queue.cc:237] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.727885 4098 raft_consensus.cc:493] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.728029 4098 raft_consensus.cc:515] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.728669 4098 leader_election.cc:290] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:51.734550 4374 raft_consensus.cc:2749] T bc4ca4423e3a4e0c96846babec931fec P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:51.737741 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_de99eb535be548db8335e7fbcfe8adb7" } new_table_name: "default.Table_4a429874a49845f9b6d8a512bf1b9680" modify_external_catalogs: true
I20251026 02:56:51.742180 2381 tablet_bootstrap.cc:492] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
W20251026 02:56:51.742308 1657 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5983c01631e9479d9f5e4ba3b7303909 consensus state has no leader
I20251026 02:56:51.742429 2381 ts_tablet_manager.cc:1403] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.409s user 0.005s sys 0.013s
I20251026 02:56:51.743225 2381 raft_consensus.cc:359] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.743523 2381 raft_consensus.cc:385] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:51.743646 2381 raft_consensus.cc:740] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:51.743995 2381 consensus_queue.cc:260] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.747016 1657 catalog_manager.cc:5649] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:51.747642 1657 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 398c1f420974453b9b6b0fe17795378b (table default.Table_a92e68d05f4b455d810180ee4cc5874d [id=7f64fbec3582415a8924d01e14856a56]). Expected version 1 got 0
I20251026 02:56:51.752179 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 014f5a300c6a4cc88a2bf50fc85680af (DEFAULT_TABLE table=default.Table_94bf0678dbe746219b474992c5acde07 [id=b3dc2a9b971f4bd2851bec852e662fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.752642 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 014f5a300c6a4cc88a2bf50fc85680af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.755787 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 28dfe285139042349b5b9d6cbc59a778
W20251026 02:56:51.758591 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 142048us
I20251026 02:56:51.758613 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 28dfe285139042349b5b9d6cbc59a778 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:51.759292 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be3ac522207d41e6915e5b9a580c6005. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.759321 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e372c60e85634713bc49acb9d2fbee32" } modify_external_catalogs: true
I20251026 02:56:51.759395 2092 tablet_service.cc:1552] Processing DeleteTablet for tablet 28dfe285139042349b5b9d6cbc59a778 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:51.759475 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_e372c60e85634713bc49acb9d2fbee32" } modify_external_catalogs: true
I20251026 02:56:51.767159 2092 tablet_service.cc:1505] Processing CreateTablet for tablet 79421734580944369cbd8ac6597996e1 (DEFAULT_TABLE table=default.Table_be9843c93f174ca385806dea17d6d181 [id=b6cb29a5864e4205a3506f79ef11c3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.767590 2092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79421734580944369cbd8ac6597996e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.770154 2100 tablet_service.cc:1505] Processing CreateTablet for tablet f1ebfd8b05ce41fc873ff0ec2c377aec (DEFAULT_TABLE table=default.Table_b52a73c60cda4bc58b800957c4f9bed1 [id=f47497d9340f451c87e618bbaacfecb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.771607 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1ebfd8b05ce41fc873ff0ec2c377aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.772879 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 79421734580944369cbd8ac6597996e1 (DEFAULT_TABLE table=default.Table_be9843c93f174ca385806dea17d6d181 [id=b6cb29a5864e4205a3506f79ef11c3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.772969 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 8498e8f29fab4d659651adf0c238aae6 (DEFAULT_TABLE table=default.Table_1c705e5c947c4376b1d4ecdd4a12ac95 [id=a4b3cfd4757a4cb497913297f8160508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.773214 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79421734580944369cbd8ac6597996e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.773200 2096 tablet_service.cc:1505] Processing CreateTablet for tablet 8498e8f29fab4d659651adf0c238aae6 (DEFAULT_TABLE table=default.Table_1c705e5c947c4376b1d4ecdd4a12ac95 [id=a4b3cfd4757a4cb497913297f8160508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.773459 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8498e8f29fab4d659651adf0c238aae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.773530 2096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8498e8f29fab4d659651adf0c238aae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.776300 4387 raft_consensus.cc:493] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.776432 4387 raft_consensus.cc:515] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.777029 4387 leader_election.cc:290] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.777714 2256 raft_consensus.cc:3060] T 1dbbfb77851d443f8cf4d798f3e07e59 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.770921 2087 tablet_service.cc:1505] Processing CreateTablet for tablet b96528939e4445e09dec2195b4302a3e (DEFAULT_TABLE table=default.Table_02cc9b3dbd8b4d43b6948f7b9eb3e566 [id=6274789835b541b8b2e0f4936ab8fafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.781342 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 04265a5178034dcaa2bb14b1efa3e493 (DEFAULT_TABLE table=default.Table_9f6ae0bb97e0495895985e078c5e79b0 [id=ee90f26d898241aab04a07df7a453f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.781731 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04265a5178034dcaa2bb14b1efa3e493. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.781803 2114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1d8bfca6a554e2abbbb9a0c7e2c0c94" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:51.781977 2114 raft_consensus.cc:2468] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:51.771209 2098 tablet_service.cc:1505] Processing CreateTablet for tablet bb40f45aebb5463b99ab53590cc10185 (DEFAULT_TABLE table=default.Table_f0758cabc30346959fcaac8df0a440a0 [id=a44f7d24f9304e9e990feafaaf2baa9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.783815 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb40f45aebb5463b99ab53590cc10185. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.787744 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b96528939e4445e09dec2195b4302a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.788800 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 4010e810fee5467db35ec065f4f0ae89 (DEFAULT_TABLE table=default.Table_ac9afcac152340fbb486392806532471 [id=5086df93e533428185465ec6150122b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.789124 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4010e810fee5467db35ec065f4f0ae89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.789762 1945 tablet_service.cc:1505] Processing CreateTablet for tablet 56b4b784c9e2403e8a3c7d31859568c0 (DEFAULT_TABLE table=default.Table_9f662442aecd4fe5a7d273b0c1e6aca6 [id=0d0d8b1c451b43e5bd8876f697bebf0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.790040 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b4b784c9e2403e8a3c7d31859568c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.792326 1959 tablet_service.cc:1552] Processing DeleteTablet for tablet 7060cb495eb1423b9e3ecbd17c325948 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:51.792701 2376 ts_tablet_manager.cc:1929] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:51.792790 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 72e4a4ea953f4f2786a39c0ddbd11219 (DEFAULT_TABLE table=default.Table_7a245074eac243e29b301b5f1c84a32d [id=755e624a8fcd4a00bc9878ed74678e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.792841 2099 tablet_service.cc:1505] Processing CreateTablet for tablet e9d0bdbe628449958cc33ae366ac81b6 (DEFAULT_TABLE table=default.Table_5e8fcf7ea82a40c4bf2e9a09300d7b97 [id=54ba56f1f8274014a4bddbfeb4c6dc0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.793031 1956 tablet_service.cc:1505] Processing CreateTablet for tablet b084395d643a4cb5b9a6f900d8a871bf (DEFAULT_TABLE table=default.Table_48fb5691d1674c79a437be2759a72637 [id=cad5aa4b07a948d4b5b720fd42468c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.793123 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e4a4ea953f4f2786a39c0ddbd11219. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.793358 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b084395d643a4cb5b9a6f900d8a871bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.793457 2089 tablet_service.cc:1505] Processing CreateTablet for tablet 232fd4092e9a4a7a9fc9200d29180157 (DEFAULT_TABLE table=default.Table_76e2e866c5b84956aac1e1bcde61692e [id=1d09f5b9fa164ac783f0be9332eac4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.793751 2089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 232fd4092e9a4a7a9fc9200d29180157. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.796478 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 527d9e1549b6472fbc2dedc21271725e (DEFAULT_TABLE table=default.Table_9520015608434b2990a378af62b56414 [id=e6aeed19469d4404bcf832f1a80af050]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.796808 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527d9e1549b6472fbc2dedc21271725e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.800984 2099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9d0bdbe628449958cc33ae366ac81b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.801748 1960 tablet_service.cc:1552] Processing DeleteTablet for tablet 28dfe285139042349b5b9d6cbc59a778 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:51.802114 1947 tablet_service.cc:1505] Processing CreateTablet for tablet 79421734580944369cbd8ac6597996e1 (DEFAULT_TABLE table=default.Table_be9843c93f174ca385806dea17d6d181 [id=b6cb29a5864e4205a3506f79ef11c3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.802490 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79421734580944369cbd8ac6597996e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.803462 2195 leader_election.cc:304] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:51.803946 4255 raft_consensus.cc:2804] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:51.804013 4255 raft_consensus.cc:493] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:51.804064 4255 raft_consensus.cc:3060] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.805519 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 2c244e735f244909b91b0d0b41eee7b4 (DEFAULT_TABLE table=default.Table_e6e2339629ed45d39e9070c3e05c109d [id=584d27528e1747d3afc32789cf243bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.805950 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c244e735f244909b91b0d0b41eee7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.806661 4255 raft_consensus.cc:515] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.807255 4255 leader_election.cc:290] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:51.807413 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 014f5a300c6a4cc88a2bf50fc85680af (DEFAULT_TABLE table=default.Table_94bf0678dbe746219b474992c5acde07 [id=b3dc2a9b971f4bd2851bec852e662fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.807595 1943 tablet_service.cc:1505] Processing CreateTablet for tablet f21dfb250b814ac28bce8fad0ec9d7bb (DEFAULT_TABLE table=default.Table_d14a01ea46f346e2aa335721052454fe [id=567f5a6f3b6641e388d577fb768ca7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.807870 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f21dfb250b814ac28bce8fad0ec9d7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.808053 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 014f5a300c6a4cc88a2bf50fc85680af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.808712 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 4703003bce6f4548b0e1bf9a3f7095f9 (DEFAULT_TABLE table=default.Table_91d47d9643cc4d25a7fb31d6360abd20 [id=34cb26ae7e2f424eac61a959d1aa5f50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.809048 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4703003bce6f4548b0e1bf9a3f7095f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.809163 2085 tablet_service.cc:1505] Processing CreateTablet for tablet cbe1046b44534c268510cd8588f12c80 (DEFAULT_TABLE table=default.Table_9c1813a4f5414f74a27525b8906a380b [id=b0ccb1388ffc4875a6c6d83a2dd0488d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.809557 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe1046b44534c268510cd8588f12c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.812505 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 57e91c7d4f734c839476596e4322738b (DEFAULT_TABLE table=default.Table_01bfed50414a4ccbba2a65baf1a055cd [id=2d6b3c5b970c49f3a009972bc3e74213]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.812863 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e91c7d4f734c839476596e4322738b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.813609 2095 tablet_service.cc:1505] Processing CreateTablet for tablet 409b4ba5b14f42cca5658a7c61ea6dee (DEFAULT_TABLE table=default.Table_a3b2085212a547c1ac7c6ad183faa9ea [id=69f91eecc591453d9486cb5314362abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.813922 2095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 409b4ba5b14f42cca5658a7c61ea6dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.813892 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1d8bfca6a554e2abbbb9a0c7e2c0c94" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:51.814055 2110 raft_consensus.cc:3060] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.815796 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7834a85168364390a841fc883681c845" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:51.816450 2097 tablet_service.cc:1505] Processing CreateTablet for tablet 89a7853ff92c4350822b205a322fc62b (DEFAULT_TABLE table=default.Table_d77bb0c25d55465c971fe532c05a5754 [id=4e0e65a9160442e2b0bd7bfb7199530c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.816782 2097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89a7853ff92c4350822b205a322fc62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.818209 2110 raft_consensus.cc:2468] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:51.820466 2083 tablet_service.cc:1505] Processing CreateTablet for tablet 224b02d432f8451da069f3ff8519c22b (DEFAULT_TABLE table=default.Table_e4bfd02ebda04e118feee745199d6bbb [id=2565a43ea08840b7b269584a47a6a05b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.820766 2083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 224b02d432f8451da069f3ff8519c22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.823040 2372 ts_tablet_manager.cc:1929] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:51.823141 2372 log.cc:1199] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/1eabfc95caa84afc9a8956a7ac13fea6
I20251026 02:56:51.823462 2372 ts_tablet_manager.cc:1950] T 1eabfc95caa84afc9a8956a7ac13fea6 P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:51.824852 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e830f1aa1270420c9b28a0491a1a0960" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.814682 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 7eb85291b9fd4fed92139084df5eed60 (DEFAULT_TABLE table=default.Table_21830ac7f7ef4cfdb68626b890c460b6 [id=2fcb2c98ea654a0da7ae2f5e8ab6b36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.824980 1977 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "868dbbfce40c4dd6b36cc220966b1986" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.816120 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 61a931473df24f97a75200450ac7f3b6 (DEFAULT_TABLE table=default.Table_d72028b7f087490dbd200e4172900e7e [id=4dba4794ab354a2e90c7102138335727]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.816190 1973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55d8d698ef0f4a81a0e269943fa41206" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.825172 1978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8fe7825fa8d46669772cf84bc4b3afc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.792785 2376 log.cc:1199] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/62d30d79a6e4432a8dbe40a104e87fa0
I20251026 02:56:51.816550 1980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00a8f0594509497a9cb856f7890da302" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.825702 2376 ts_tablet_manager.cc:1950] T 62d30d79a6e4432a8dbe40a104e87fa0 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:51.816706 1976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1d8bfca6a554e2abbbb9a0c7e2c0c94" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.816825 1979 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7834a85168364390a841fc883681c845" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.819458 1966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3af2695c41a4439bc4092b7cb1b2edf" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.825948 1974 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "142043b5d558490c9399626c3764ff7e" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.826017 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1072) took 3467 ms. Trace:
I20251026 02:56:51.826090 2376 rpcz_store.cc:276] 1026 02:56:48.358639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.358821 (+ 182us) service_pool.cc:225] Handling call
1026 02:56:51.583520 (+3224699us) ts_tablet_manager.cc:1151] Deleting tablet 62d30d79a6e4432a8dbe40a104e87fa0
1026 02:56:51.583526 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.595965 (+ 12439us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.792625 (+196660us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.826001 (+ 33376us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3224436}
I20251026 02:56:51.818612 1982 raft_consensus.cc:3060] T 3b77bb6b4722449c9699b548a9ef6314 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:51.826361 2376 tablet_replica.cc:333] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:51.826455 2093 tablet_service.cc:1505] Processing CreateTablet for tablet aedea0fe15f941cda21fea3dab978d91 (DEFAULT_TABLE table=default.Table_5d7cf550840a4c5780ef193e148b71ec [id=432434e61ef641448fe7ba1fba2ddcb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.826555 2376 raft_consensus.cc:2243] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:51.819604 1965 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3af2695c41a4439bc4092b7cb1b2edf" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:51.823972 1968 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8fe7825fa8d46669772cf84bc4b3afc" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:51.826686 2376 raft_consensus.cc:2272] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:51.824463 1970 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1d8bfca6a554e2abbbb9a0c7e2c0c94" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:51.826777 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aedea0fe15f941cda21fea3dab978d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.820778 2375 ts_tablet_manager.cc:1434] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.118s user 0.002s sys 0.000s
I20251026 02:56:51.830228 2375 tablet_bootstrap.cc:492] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:51.832469 2375 tablet_bootstrap.cc:654] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:51.832544 2376 ts_tablet_manager.cc:1916] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:51.832593 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1d8bfca6a554e2abbbb9a0c7e2c0c94" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:51.832902 2257 raft_consensus.cc:2393] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b6b0a6c4e144982b67980118cddba48 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:51.833360 1966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1d8bfca6a554e2abbbb9a0c7e2c0c94" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:51.833921 2082 tablet_service.cc:1505] Processing CreateTablet for tablet 61a931473df24f97a75200450ac7f3b6 (DEFAULT_TABLE table=default.Table_d72028b7f087490dbd200e4172900e7e [id=4dba4794ab354a2e90c7102138335727]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.834237 2082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61a931473df24f97a75200450ac7f3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.835739 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61a931473df24f97a75200450ac7f3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.836076 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb85291b9fd4fed92139084df5eed60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.848871 2241 tablet_service.cc:1505] Processing CreateTablet for tablet f1ebfd8b05ce41fc873ff0ec2c377aec (DEFAULT_TABLE table=default.Table_b52a73c60cda4bc58b800957c4f9bed1 [id=f47497d9340f451c87e618bbaacfecb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.849378 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1ebfd8b05ce41fc873ff0ec2c377aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.852993 2381 ts_tablet_manager.cc:1434] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.110s user 0.001s sys 0.001s
I20251026 02:56:51.853875 2381 tablet_bootstrap.cc:492] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:51.859118 2053 leader_election.cc:343] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.859741 2237 tablet_service.cc:1505] Processing CreateTablet for tablet bb40f45aebb5463b99ab53590cc10185 (DEFAULT_TABLE table=default.Table_f0758cabc30346959fcaac8df0a440a0 [id=a44f7d24f9304e9e990feafaaf2baa9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.860823 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb40f45aebb5463b99ab53590cc10185. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.693974 2377 log.cc:1199] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/c2d6691bb4cf4a8292b0c082e54a91a5
I20251026 02:56:51.864674 2377 ts_tablet_manager.cc:1950] T c2d6691bb4cf4a8292b0c082e54a91a5 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:51.865455 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1161) took 2886 ms. Trace:
I20251026 02:56:51.865600 2377 rpcz_store.cc:276] 1026 02:56:48.978753 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.979128 (+ 375us) service_pool.cc:225] Handling call
1026 02:56:51.544964 (+2565836us) ts_tablet_manager.cc:1151] Deleting tablet c2d6691bb4cf4a8292b0c082e54a91a5
1026 02:56:51.544975 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.690840 (+145865us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.693761 (+ 2921us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.865432 (+171671us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2565729}
I20251026 02:56:51.866158 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1163) took 2887 ms. Trace:
I20251026 02:56:51.866276 2377 rpcz_store.cc:276] 1026 02:56:48.978911 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.979079 (+ 168us) service_pool.cc:225] Handling call
1026 02:56:51.866041 (+2886962us) ts_tablet_manager.cc:1151] Deleting tablet 8306f2ece961480d906df88f40b9740b
1026 02:56:51.866050 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.866142 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2886671}
I20251026 02:56:51.866684 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1159) took 2888 ms. Trace:
I20251026 02:56:51.866796 2377 rpcz_store.cc:276] 1026 02:56:48.978611 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.982379 (+ 3768us) service_pool.cc:225] Handling call
1026 02:56:51.866610 (+2884231us) ts_tablet_manager.cc:1151] Deleting tablet c31d2704d1f14ddead69c794cc631ec1
1026 02:56:51.866619 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.866671 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2884071}
I20251026 02:56:51.870326 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 2ac6e19b04974fd585822340c0bf6169 (DEFAULT_TABLE table=default.Table_d89900bf7bf04855a2bd970460c420e8 [id=fabca86bb32941c7883bd8e9e6a239aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.871243 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ac6e19b04974fd585822340c0bf6169. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.875175 1940 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58886 (request call id 189) took 1713 ms. Trace:
I20251026 02:56:51.875353 1940 rpcz_store.cc:276] 1026 02:56:50.161620 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:50.195985 (+ 34365us) service_pool.cc:225] Handling call
1026 02:56:51.158867 (+962882us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x61600012a054. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:51.875151 (+716284us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":736928512}
I20251026 02:56:51.887217 1939 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58886 (request call id 192) took 1093 ms. Trace:
I20251026 02:56:51.887409 1939 rpcz_store.cc:276] 1026 02:56:50.793301 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:50.793361 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:50.848548 (+ 55187us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x6160001a4454. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:51.390301 (+541753us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x6160000e5954. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:51.487334 (+ 97033us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x6160000f8e54. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:51.887186 (+399852us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":577707520}
W20251026 02:56:51.893931 1658 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 106524801a204a81b9f194321768f433 consensus state has no leader
I20251026 02:56:51.868086 2232 tablet_service.cc:1505] Processing CreateTablet for tablet e9d0bdbe628449958cc33ae366ac81b6 (DEFAULT_TABLE table=default.Table_5e8fcf7ea82a40c4bf2e9a09300d7b97 [id=54ba56f1f8274014a4bddbfeb4c6dc0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.868535 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 232fd4092e9a4a7a9fc9200d29180157 (DEFAULT_TABLE table=default.Table_76e2e866c5b84956aac1e1bcde61692e [id=1d09f5b9fa164ac783f0be9332eac4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.868992 2235 tablet_service.cc:1505] Processing CreateTablet for tablet 527d9e1549b6472fbc2dedc21271725e (DEFAULT_TABLE table=default.Table_9520015608434b2990a378af62b56414 [id=e6aeed19469d4404bcf832f1a80af050]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.869093 2233 tablet_service.cc:1505] Processing CreateTablet for tablet c38f7a4543de40709085198e6fc590ff (DEFAULT_TABLE table=default.Table_34265ddfd85f4788b015e22efaf25d02 [id=d11587d6977d498d9e1a009a80371181]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.869306 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 72e4a4ea953f4f2786a39c0ddbd11219 (DEFAULT_TABLE table=default.Table_7a245074eac243e29b301b5f1c84a32d [id=755e624a8fcd4a00bc9878ed74678e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.888609 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 7eb85291b9fd4fed92139084df5eed60 (DEFAULT_TABLE table=default.Table_21830ac7f7ef4cfdb68626b890c460b6 [id=2fcb2c98ea654a0da7ae2f5e8ab6b36c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:51.892525 2194 consensus_peers.cc:597] T 109073eb664d4090b7a2b786146e13fa P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:51.899080 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 232fd4092e9a4a7a9fc9200d29180157. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:51.899240 2194 leader_election.cc:343] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.899361 2194 leader_election.cc:304] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:51.899720 2194 leader_election.cc:343] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.900249 4387 raft_consensus.cc:2749] T 868dbbfce40c4dd6b36cc220966b1986 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:51.900411 4387 consensus_queue.cc:1048] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:51.902071 2194 leader_election.cc:343] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:51.902916 2194 leader_election.cc:343] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:51.903275 2194 leader_election.cc:343] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:51.903576 2194 consensus_peers.cc:597] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:51.903738 2194 leader_election.cc:343] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:51.905537 2194 leader_election.cc:343] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.870025 4098 raft_consensus.cc:493] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:51.906133 4098 raft_consensus.cc:515] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.906714 4098 leader_election.cc:290] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:51.906932 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 47838.2us
W20251026 02:56:51.908165 2053 leader_election.cc:343] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.908399 2053 leader_election.cc:304] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:51.908671 4098 raft_consensus.cc:2749] T e830f1aa1270420c9b28a0491a1a0960 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:51.909036 2053 leader_election.cc:343] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.909185 2053 leader_election.cc:304] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:51.909430 4098 raft_consensus.cc:2749] T 00a8f0594509497a9cb856f7890da302 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:51.909822 2053 leader_election.cc:343] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:51.913026 2378 tablet_bootstrap.cc:492] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:51.913210 2378 ts_tablet_manager.cc:1403] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.661s user 0.014s sys 0.006s
I20251026 02:56:51.915423 2235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527d9e1549b6472fbc2dedc21271725e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.920390 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1162) took 2941 ms. Trace:
I20251026 02:56:51.920481 2377 rpcz_store.cc:276] 1026 02:56:48.978827 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.982572 (+ 3745us) service_pool.cc:225] Handling call
1026 02:56:51.898582 (+2916010us) ts_tablet_manager.cc:1151] Deleting tablet 21b936fa4f6a4ff6af321971a30cccbe
1026 02:56:51.920266 (+ 21684us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.920363 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2915887}
I20251026 02:56:51.920475 2233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c38f7a4543de40709085198e6fc590ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.921257 2231 tablet_service.cc:1505] Processing CreateTablet for tablet aedea0fe15f941cda21fea3dab978d91 (DEFAULT_TABLE table=default.Table_5d7cf550840a4c5780ef193e148b71ec [id=432434e61ef641448fe7ba1fba2ddcb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.921650 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aedea0fe15f941cda21fea3dab978d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.920957 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 57e91c7d4f734c839476596e4322738b (DEFAULT_TABLE table=default.Table_01bfed50414a4ccbba2a65baf1a055cd [id=2d6b3c5b970c49f3a009972bc3e74213]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.924399 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e4a4ea953f4f2786a39c0ddbd11219. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.924646 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e91c7d4f734c839476596e4322738b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.920957 2234 tablet_service.cc:1505] Processing CreateTablet for tablet cbe1046b44534c268510cd8588f12c80 (DEFAULT_TABLE table=default.Table_9c1813a4f5414f74a27525b8906a380b [id=b0ccb1388ffc4875a6c6d83a2dd0488d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.921257 2377 tablet_replica.cc:333] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:51.924963 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe1046b44534c268510cd8588f12c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.920783 2240 tablet_service.cc:1505] Processing CreateTablet for tablet 4703003bce6f4548b0e1bf9a3f7095f9 (DEFAULT_TABLE table=default.Table_91d47d9643cc4d25a7fb31d6360abd20 [id=34cb26ae7e2f424eac61a959d1aa5f50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.928725 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4703003bce6f4548b0e1bf9a3f7095f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.930256 2377 raft_consensus.cc:2243] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:51.930557 2377 pending_rounds.cc:70] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539: Trying to abort 1 pending ops.
I20251026 02:56:51.930600 2377 pending_rounds.cc:77] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214888586071859200 op_type: NO_OP noop_request { }
I20251026 02:56:51.930676 2377 raft_consensus.cc:2889] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 02:56:51.930759 2377 raft_consensus.cc:2272] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:51.931288 2377 ts_tablet_manager.cc:1916] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:51.933907 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 409b4ba5b14f42cca5658a7c61ea6dee (DEFAULT_TABLE table=default.Table_a3b2085212a547c1ac7c6ad183faa9ea [id=69f91eecc591453d9486cb5314362abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.934789 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 409b4ba5b14f42cca5658a7c61ea6dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.936892 1661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_fa22c9dc29814d9cb5c4bfdec072a3cf" } new_table_name: "default.Table_d3c03ddb982243a0be489ba162c0e80f" modify_external_catalogs: true
I20251026 02:56:51.938191 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_064d2f38544443f4aee739f470193114"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.938652 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_064d2f38544443f4aee739f470193114 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.944859 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9d0bdbe628449958cc33ae366ac81b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.955650 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb85291b9fd4fed92139084df5eed60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.959335 2377 ts_tablet_manager.cc:1929] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:51.959470 2377 log.cc:1199] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/dfd3e23d83a84648bd11cd258377aecb
W20251026 02:56:51.959960 1647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.60:33729, user_credentials={real_user=slave}} blocked reactor thread for 73151.9us
I20251026 02:56:51.960413 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 62d30d79a6e4432a8dbe40a104e87fa0 (table default.Table_5f7b25fab37c4983acf922bd78374da6 [id=777528a5bf944ecb96ebdbea9e7ee559]) successfully deleted
I20251026 02:56:51.962834 2195 leader_election.cc:304] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:51.963096 4069 raft_consensus.cc:2804] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:51.963099 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 89a7853ff92c4350822b205a322fc62b (DEFAULT_TABLE table=default.Table_d77bb0c25d55465c971fe532c05a5754 [id=4e0e65a9160442e2b0bd7bfb7199530c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.963256 4069 raft_consensus.cc:697] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:51.963459 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89a7853ff92c4350822b205a322fc62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.963482 4069 consensus_queue.cc:237] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.965641 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 224b02d432f8451da069f3ff8519c22b (DEFAULT_TABLE table=default.Table_e4bfd02ebda04e118feee745199d6bbb [id=2565a43ea08840b7b269584a47a6a05b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.966051 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 224b02d432f8451da069f3ff8519c22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.894483 2375 tablet_bootstrap.cc:492] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:51.970921 2053 leader_election.cc:304] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:51.932677 2088 tablet_service.cc:1505] Processing CreateTablet for tablet c38f7a4543de40709085198e6fc590ff (DEFAULT_TABLE table=default.Table_34265ddfd85f4788b015e22efaf25d02 [id=d11587d6977d498d9e1a009a80371181]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.934593 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 7eb85291b9fd4fed92139084df5eed60 (DEFAULT_TABLE table=default.Table_21830ac7f7ef4cfdb68626b890c460b6 [id=2fcb2c98ea654a0da7ae2f5e8ab6b36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.971058 2375 ts_tablet_manager.cc:1403] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.141s user 0.003s sys 0.000s
I20251026 02:56:51.917177 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 047bfb3525054c44aeb7f37625d9d8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.972294 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb85291b9fd4fed92139084df5eed60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.975630 4098 raft_consensus.cc:2749] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:51.976500 2375 raft_consensus.cc:359] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.976773 2375 raft_consensus.cc:385] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:51.976862 2375 raft_consensus.cc:740] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:51.977123 2375 consensus_queue.cc:260] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:51.977809 2375 ts_tablet_manager.cc:1434] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251026 02:56:51.978121 2375 tablet_bootstrap.cc:492] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:51.978421 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c38f7a4543de40709085198e6fc590ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.978863 1659 master_service.cc:954] ReplaceTablet: received request to replace tablet cffa8c7cf71a4d449803e163b415f473 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:51.980093 2375 tablet_bootstrap.cc:654] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:51.981274 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 71446.4us
I20251026 02:56:51.982110 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a43c7d16330644b5bfa0a88e5b624f6b
W20251026 02:56:51.982584 4392 log.cc:927] Time spent T e203fe0403594d30b44d531153d42a10 P 7fef01f4f2b843538ea245a09dd087a8: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
W20251026 02:56:51.984360 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6403483a7e0f4676a8a88ce55beeae80 consensus state has no leader
I20251026 02:56:51.986204 2377 ts_tablet_manager.cc:1950] T dfd3e23d83a84648bd11cd258377aecb P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:51.988353 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 8498e8f29fab4d659651adf0c238aae6 (DEFAULT_TABLE table=default.Table_1c705e5c947c4376b1d4ecdd4a12ac95 [id=a4b3cfd4757a4cb497913297f8160508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.988713 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8498e8f29fab4d659651adf0c238aae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.989120 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_43fe592c9103480e8fb16e77712556d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.989741 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43fe592c9103480e8fb16e77712556d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.990984 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 03eb7f47578247a48a885489730d527e
I20251026 02:56:51.991451 1947 tablet_service.cc:1505] Processing CreateTablet for tablet f1ebfd8b05ce41fc873ff0ec2c377aec (DEFAULT_TABLE table=default.Table_b52a73c60cda4bc58b800957c4f9bed1 [id=f47497d9340f451c87e618bbaacfecb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.991796 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1ebfd8b05ce41fc873ff0ec2c377aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:51.991802 1654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4ad9ab3f7c6c4ad2b11fc443473ab58e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:51.992362 1654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ad9ab3f7c6c4ad2b11fc443473ab58e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:51.993698 1662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_63a0dc3ab9a14e37b44c2d6c776ca543" } modify_external_catalogs: true
I20251026 02:56:51.993857 1662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_63a0dc3ab9a14e37b44c2d6c776ca543" } modify_external_catalogs: true
I20251026 02:56:51.994009 1945 tablet_service.cc:1505] Processing CreateTablet for tablet b96528939e4445e09dec2195b4302a3e (DEFAULT_TABLE table=default.Table_02cc9b3dbd8b4d43b6948f7b9eb3e566 [id=6274789835b541b8b2e0f4936ab8fafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.994385 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2bb5a8427c5044cc8dc28649dea1f4b1" } modify_external_catalogs: true
I20251026 02:56:51.994506 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2bb5a8427c5044cc8dc28649dea1f4b1" } modify_external_catalogs: true
I20251026 02:56:51.996043 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_22ed9a263e5942e4b6fdd13731e49e1d" } new_table_name: "default.Table_991a5e4bffdd4dc8b35db5027c582129" modify_external_catalogs: true
I20251026 02:56:51.997042 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 527d9e1549b6472fbc2dedc21271725e (DEFAULT_TABLE table=default.Table_9520015608434b2990a378af62b56414 [id=e6aeed19469d4404bcf832f1a80af050]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.998240 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 72e4a4ea953f4f2786a39c0ddbd11219 (DEFAULT_TABLE table=default.Table_7a245074eac243e29b301b5f1c84a32d [id=755e624a8fcd4a00bc9878ed74678e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.998778 1951 tablet_service.cc:1505] Processing CreateTablet for tablet e9d0bdbe628449958cc33ae366ac81b6 (DEFAULT_TABLE table=default.Table_5e8fcf7ea82a40c4bf2e9a09300d7b97 [id=54ba56f1f8274014a4bddbfeb4c6dc0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.998945 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 232fd4092e9a4a7a9fc9200d29180157 (DEFAULT_TABLE table=default.Table_76e2e866c5b84956aac1e1bcde61692e [id=1d09f5b9fa164ac783f0be9332eac4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:51.998778 1954 tablet_service.cc:1505] Processing CreateTablet for tablet c38f7a4543de40709085198e6fc590ff (DEFAULT_TABLE table=default.Table_34265ddfd85f4788b015e22efaf25d02 [id=d11587d6977d498d9e1a009a80371181]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:52.000068 1661 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6294c8954f9749f998d5ae1c856e02ff consensus state has no leader
I20251026 02:56:52.000919 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_379aca7c219b4cbcbdba213c3ade48d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:52.001550 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_379aca7c219b4cbcbdba213c3ade48d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:52.005225 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3e17f384354f44e48cdfa79c4cac560d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:52.005937 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e17f384354f44e48cdfa79c4cac560d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:52.010445 2378 raft_consensus.cc:359] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.010712 2378 raft_consensus.cc:385] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.010807 2378 raft_consensus.cc:740] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.011066 2378 consensus_queue.cc:260] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.016196 2237 tablet_service.cc:1505] Processing CreateTablet for tablet b96528939e4445e09dec2195b4302a3e (DEFAULT_TABLE table=default.Table_02cc9b3dbd8b4d43b6948f7b9eb3e566 [id=6274789835b541b8b2e0f4936ab8fafc]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:52.020661 2053 leader_election.cc:343] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.021166 1943 tablet_service.cc:1505] Processing CreateTablet for tablet bb40f45aebb5463b99ab53590cc10185 (DEFAULT_TABLE table=default.Table_f0758cabc30346959fcaac8df0a440a0 [id=a44f7d24f9304e9e990feafaaf2baa9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.021646 1659 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet cffa8c7cf71a4d449803e163b415f473
I20251026 02:56:52.022214 1659 catalog_manager.cc:6719] ReplaceTablet: tablet cffa8c7cf71a4d449803e163b415f473 deleted and replaced by tablet 01b2eeedec7f48088b54d79394432dd7
I20251026 02:56:52.024505 1945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b96528939e4445e09dec2195b4302a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.028122 2376 ts_tablet_manager.cc:1929] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:52.028997 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527d9e1549b6472fbc2dedc21271725e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.030577 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e4a4ea953f4f2786a39c0ddbd11219. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.030968 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9d0bdbe628449958cc33ae366ac81b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.031173 1943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb40f45aebb5463b99ab53590cc10185. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.031543 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c38f7a4543de40709085198e6fc590ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.032083 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 232fd4092e9a4a7a9fc9200d29180157. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.034853 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d77bb0c25d55465c971fe532c05a5754" } modify_external_catalogs: true
I20251026 02:56:52.035090 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_d77bb0c25d55465c971fe532c05a5754" } modify_external_catalogs: true
I20251026 02:56:52.035866 1655 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_52417f25dbdb4676a5aaaca9bf46962c" } new_table_name: "default.Table_6292b26811764e51a3d7f831c6f60b7d" modify_external_catalogs: true
I20251026 02:56:52.042814 1659 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet cfac07917ee34e6dba3fac5e081f6b43 (table default.Table_907991c2b0034c118065ead079f273f9 [id=a8d490f3dbc948d38c54d14e178b736e]). Expected version 1 got 0
I20251026 02:56:52.043031 1659 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet 41a1442fffe54add93666e3c78a0cc58 (table default.Table_1faa0c8151724c5e816546ee2e1d5182 [id=a106f9ece0b9467a8bc1d5d4b3524c41]). Expected version 1 got 0
I20251026 02:56:52.052423 2376 log.cc:1199] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/9c7efb70c0804c149f36440a4a9e6ef8
I20251026 02:56:52.057233 2378 ts_tablet_manager.cc:1434] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.144s user 0.001s sys 0.001s
I20251026 02:56:52.018044 4387 consensus_queue.cc:1048] T 85420a102dae4269811dbdc64aebb20a P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:52.019053 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1172) took 2763 ms. Trace:
I20251026 02:56:52.057951 2377 rpcz_store.cc:276] 1026 02:56:49.255441 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:49.258111 (+ 2670us) service_pool.cc:225] Handling call
1026 02:56:51.921218 (+2663107us) ts_tablet_manager.cc:1151] Deleting tablet dfd3e23d83a84648bd11cd258377aecb
1026 02:56:51.921226 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.934064 (+ 12838us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.959248 (+ 25184us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.019033 (+ 59785us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31591,"tablet-delete.queue_time_us":2615316}
I20251026 02:56:52.058468 2377 tablet_replica.cc:333] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:52.058728 2377 raft_consensus.cc:2243] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:52.058923 2377 raft_consensus.cc:2272] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:52.059541 2377 ts_tablet_manager.cc:1916] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:52.020298 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 48458.3us
I20251026 02:56:52.069800 2237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b96528939e4445e09dec2195b4302a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.008738 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet 03eb7f47578247a48a885489730d527e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:51.983503 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet a43c7d16330644b5bfa0a88e5b624f6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:52.023010 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet cffa8c7cf71a4d449803e163b415f473 with delete_type TABLET_DATA_DELETED (replaced by tablet 01b2eeedec7f48088b54d79394432dd7) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:51.893465 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1074) took 5293 ms. Trace:
I20251026 02:56:52.070798 2372 rpcz_store.cc:276] 1026 02:56:46.599671 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.697466 (+ 97795us) service_pool.cc:225] Handling call
1026 02:56:51.407684 (+4710218us) ts_tablet_manager.cc:1151] Deleting tablet 1eabfc95caa84afc9a8956a7ac13fea6
1026 02:56:51.407699 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.714905 (+307206us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.822925 (+108020us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:51.893440 (+ 70515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4687030}
I20251026 02:56:52.071419 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 1eabfc95caa84afc9a8956a7ac13fea6 (table default.Table_0ea323ca655f45aa94b47eab713ebbd0 [id=1d82a5d03031489fa756df9cfb380b8e]) successfully deleted
I20251026 02:56:52.072257 2053 leader_election.cc:304] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:52.073705 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet c2d6691bb4cf4a8292b0c082e54a91a5 (table default.Table_8602d75889ea4576ae575b55fe95e940 [id=b0104213af32435589ae19109b7aee7f]) successfully deleted
I20251026 02:56:51.892728 2381 tablet_bootstrap.cc:654] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.080376 4098 raft_consensus.cc:2749] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:52.072831 2372 tablet_replica.cc:333] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:52.080870 2372 raft_consensus.cc:2243] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:52.082067 2372 raft_consensus.cc:2272] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:52.081259 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 60611.4us
W20251026 02:56:52.082916 2053 leader_election.cc:343] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:52.107589 1629 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 162326us
I20251026 02:56:52.108306 2053 leader_election.cc:304] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:52.108814 4098 raft_consensus.cc:2749] T 142043b5d558490c9399626c3764ff7e P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:52.109545 2053 leader_election.cc:343] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:52.111515 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 8306f2ece961480d906df88f40b9740b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8306f2ece961480d906df88f40b9740b already in progress: creating tablet
I20251026 02:56:52.111850 2377 ts_tablet_manager.cc:1929] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 02:56:52.112068 2377 log.cc:1199] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/9b9a5daa36b547a7888d38dc3329eaf2
I20251026 02:56:52.115383 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31b6e08597bd450eb1875201cf594a82" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:52.115383 2229 tablet_service.cc:1552] Processing DeleteTablet for tablet a43c7d16330644b5bfa0a88e5b624f6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:52.115515 2241 tablet_service.cc:1552] Processing DeleteTablet for tablet cffa8c7cf71a4d449803e163b415f473 with delete_type TABLET_DATA_DELETED (replaced by tablet 01b2eeedec7f48088b54d79394432dd7) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:52.115449 2230 tablet_service.cc:1552] Processing DeleteTablet for tablet 03eb7f47578247a48a885489730d527e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:52.134500 2377 ts_tablet_manager.cc:1950] T 9b9a5daa36b547a7888d38dc3329eaf2 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:52.135411 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1171) took 2880 ms. Trace:
W20251026 02:56:52.136039 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet c31d2704d1f14ddead69c794cc631ec1 because the tablet was not found. No further retry: Not found: Tablet not found: c31d2704d1f14ddead69c794cc631ec1
W20251026 02:56:52.136368 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 21b936fa4f6a4ff6af321971a30cccbe because the tablet was not found. No further retry: Not found: Tablet not found: 21b936fa4f6a4ff6af321971a30cccbe
I20251026 02:56:52.137784 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet dfd3e23d83a84648bd11cd258377aecb (table default.Table_f4bddfca37d64369b99a1fdc0102ca35 [id=434c941e53194bcf9a53a590775d6a6d]) successfully deleted
I20251026 02:56:52.138063 2053 leader_election.cc:304] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:52.138475 4098 raft_consensus.cc:2749] T 55d8d698ef0f4a81a0e269943fa41206 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:52.107246 2372 ts_tablet_manager.cc:1916] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:52.148012 2375 tablet_bootstrap.cc:492] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:52.148245 2375 ts_tablet_manager.cc:1403] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.170s user 0.005s sys 0.000s
I20251026 02:56:52.149345 2375 raft_consensus.cc:359] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.154201 1960 tablet_service.cc:1505] Processing CreateTablet for tablet 57e91c7d4f734c839476596e4322738b (DEFAULT_TABLE table=default.Table_01bfed50414a4ccbba2a65baf1a055cd [id=2d6b3c5b970c49f3a009972bc3e74213]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.154536 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e91c7d4f734c839476596e4322738b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.155465 1952 tablet_service.cc:1505] Processing CreateTablet for tablet aedea0fe15f941cda21fea3dab978d91 (DEFAULT_TABLE table=default.Table_5d7cf550840a4c5780ef193e148b71ec [id=432434e61ef641448fe7ba1fba2ddcb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.155742 1952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aedea0fe15f941cda21fea3dab978d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.156311 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 409b4ba5b14f42cca5658a7c61ea6dee (DEFAULT_TABLE table=default.Table_a3b2085212a547c1ac7c6ad183faa9ea [id=69f91eecc591453d9486cb5314362abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.158246 2378 tablet_bootstrap.cc:492] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:52.161936 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet a43c7d16330644b5bfa0a88e5b624f6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:52.162111 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet cffa8c7cf71a4d449803e163b415f473 with delete_type TABLET_DATA_DELETED (replaced by tablet 01b2eeedec7f48088b54d79394432dd7) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:52.162262 1949 tablet_service.cc:1552] Processing DeleteTablet for tablet 03eb7f47578247a48a885489730d527e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:52.162358 1949 tablet_service.cc:1552] Processing DeleteTablet for tablet bc4ca4423e3a4e0c96846babec931fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:52.162441 1949 tablet_service.cc:1552] Processing DeleteTablet for tablet 62d30d79a6e4432a8dbe40a104e87fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:38 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:52.163358 1948 tablet_service.cc:1505] Processing CreateTablet for tablet 89a7853ff92c4350822b205a322fc62b (DEFAULT_TABLE table=default.Table_d77bb0c25d55465c971fe532c05a5754 [id=4e0e65a9160442e2b0bd7bfb7199530c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.163688 1948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89a7853ff92c4350822b205a322fc62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.164283 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 409b4ba5b14f42cca5658a7c61ea6dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.161314 1964 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31b6e08597bd450eb1875201cf594a82" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:52.176252 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 224b02d432f8451da069f3ff8519c22b (DEFAULT_TABLE table=default.Table_e4bfd02ebda04e118feee745199d6bbb [id=2565a43ea08840b7b269584a47a6a05b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.176713 1950 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ea364c4a5cf4f3f99d9ff6904add697 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:32 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:52.184731 1958 tablet_service.cc:1505] Processing CreateTablet for tablet cbe1046b44534c268510cd8588f12c80 (DEFAULT_TABLE table=default.Table_9c1813a4f5414f74a27525b8906a380b [id=b0ccb1388ffc4875a6c6d83a2dd0488d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.153905 1947 tablet_service.cc:1505] Processing CreateTablet for tablet 4703003bce6f4548b0e1bf9a3f7095f9 (DEFAULT_TABLE table=default.Table_91d47d9643cc4d25a7fb31d6360abd20 [id=34cb26ae7e2f424eac61a959d1aa5f50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.201508 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe1046b44534c268510cd8588f12c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.204418 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4703003bce6f4548b0e1bf9a3f7095f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.210613 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 224b02d432f8451da069f3ff8519c22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.213984 2376 ts_tablet_manager.cc:1950] T 9c7efb70c0804c149f36440a4a9e6ef8 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:52.214378 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1073) took 3855 ms. Trace:
I20251026 02:56:52.214515 2376 rpcz_store.cc:276] 1026 02:56:48.358760 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.358821 (+ 61us) service_pool.cc:225] Handling call
1026 02:56:51.826324 (+3467503us) ts_tablet_manager.cc:1151] Deleting tablet 9c7efb70c0804c149f36440a4a9e6ef8
1026 02:56:51.826332 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:51.857697 (+ 31365us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.028009 (+170312us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.214363 (+186354us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17,"tablet-delete.queue_time_us":3467233}
I20251026 02:56:52.215384 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1078) took 3339 ms. Trace:
I20251026 02:56:52.215505 2376 rpcz_store.cc:276] 1026 02:56:48.875792 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.875862 (+ 70us) service_pool.cc:225] Handling call
1026 02:56:52.215258 (+3339396us) ts_tablet_manager.cc:1151] Deleting tablet abf58185766448d1a7c4f3d304b0c5b1
1026 02:56:52.215275 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.215368 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3339176}
I20251026 02:56:52.216027 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1080) took 3340 ms. Trace:
I20251026 02:56:52.216147 2376 rpcz_store.cc:276] 1026 02:56:48.876027 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.876076 (+ 49us) service_pool.cc:225] Handling call
1026 02:56:52.215935 (+3339859us) ts_tablet_manager.cc:1151] Deleting tablet 163b8f785675474e90ff90a1fe6fc056
1026 02:56:52.215943 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.216014 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3339732}
I20251026 02:56:52.216617 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1079) took 3340 ms. Trace:
I20251026 02:56:52.216735 2376 rpcz_store.cc:276] 1026 02:56:48.875926 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.876113 (+ 187us) service_pool.cc:225] Handling call
1026 02:56:52.216530 (+3340417us) ts_tablet_manager.cc:1151] Deleting tablet b2eab97dc2b9409697589a34e9cc7d6c
1026 02:56:52.216539 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.216604 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3340322}
I20251026 02:56:52.217173 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1081) took 3341 ms. Trace:
I20251026 02:56:52.217374 2376 rpcz_store.cc:276] 1026 02:56:48.876131 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.876251 (+ 120us) service_pool.cc:225] Handling call
1026 02:56:52.217088 (+3340837us) ts_tablet_manager.cc:1151] Deleting tablet 19016e8e11cc42c69ce44a819db9e298
1026 02:56:52.217096 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.217160 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3340727}
I20251026 02:56:52.217904 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1082) took 3341 ms. Trace:
I20251026 02:56:52.218027 2376 rpcz_store.cc:276] 1026 02:56:48.876924 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.877084 (+ 160us) service_pool.cc:225] Handling call
1026 02:56:52.217805 (+3340721us) ts_tablet_manager.cc:1151] Deleting tablet 551cc0916a21463d9190d14a31e79988
1026 02:56:52.217815 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.217890 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3340543}
I20251026 02:56:52.223654 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1086) took 3277 ms. Trace:
I20251026 02:56:52.223948 2376 rpcz_store.cc:276] 1026 02:56:48.945777 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.948405 (+ 2628us) service_pool.cc:225] Handling call
1026 02:56:52.223421 (+3275016us) ts_tablet_manager.cc:1151] Deleting tablet fe27c85771964e4ca0ea747bbb4826df
1026 02:56:52.223436 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.223624 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3274693}
I20251026 02:56:52.224409 2376 tablet_replica.cc:333] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:52.224726 2376 raft_consensus.cc:2243] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:52.224984 2376 raft_consensus.cc:2272] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:52.225708 2376 ts_tablet_manager.cc:1916] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:52.282296 1938 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58886 (request call id 190) took 1579 ms. Trace:
I20251026 02:56:52.282536 1938 rpcz_store.cc:276] 1026 02:56:50.702465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:50.702519 (+ 54us) service_pool.cc:225] Handling call
1026 02:56:52.282244 (+1579725us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 02:56:52.297400 1941 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58886 (request call id 191) took 1518 ms. Trace:
I20251026 02:56:52.304308 2377 rpcz_store.cc:276] 1026 02:56:49.255356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:49.255565 (+ 209us) service_pool.cc:225] Handling call
1026 02:56:52.058408 (+2802843us) ts_tablet_manager.cc:1151] Deleting tablet 9b9a5daa36b547a7888d38dc3329eaf2
1026 02:56:52.058420 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.063505 (+ 5085us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.111713 (+ 48208us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.135391 (+ 23678us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2752401}
I20251026 02:56:52.145025 2381 tablet_bootstrap.cc:492] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:52.165000 4258 consensus_queue.cc:1048] T 3b77bb6b4722449c9699b548a9ef6314 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:52.219926 2194 consensus_peers.cc:597] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:52.305159 2194 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 85273.6us
I20251026 02:56:52.309355 2381 ts_tablet_manager.cc:1403] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.456s user 0.001s sys 0.003s
I20251026 02:56:52.310528 2381 raft_consensus.cc:359] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.310858 2381 raft_consensus.cc:385] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.310961 2381 raft_consensus.cc:740] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.311326 2381 consensus_queue.cc:260] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.334348 1941 rpcz_store.cc:276] 1026 02:56:50.778476 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:50.778547 (+ 71us) service_pool.cc:225] Handling call
1026 02:56:51.223551 (+445004us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x616000136954. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:51.993659 (+770108us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x6160000f8e54. stack: 00007fc41a18f9f6 00007fc41a18f7b0 00007fc41916b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc426024200 00007fc429ed066f 00007fc4242fcc36 00007fc4242fc965 00007fc42112b27e 00007fc42112a15e 00007fc421130395 00007fc421130cbf 00007fc421130ae8 0000000000a8867c
1026 02:56:52.297366 (+303707us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1436320000}
I20251026 02:56:52.300474 2378 tablet_bootstrap.cc:654] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.347163 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1173) took 3091 ms. Trace:
I20251026 02:56:52.347440 2377 rpcz_store.cc:276] 1026 02:56:49.255527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:49.262128 (+ 6601us) service_pool.cc:225] Handling call
1026 02:56:52.304901 (+3042773us) ts_tablet_manager.cc:1151] Deleting tablet ec13c5386abe414ba25589e03d5b850a
1026 02:56:52.304918 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.347130 (+ 42212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2998836}
I20251026 02:56:52.347975 2377 tablet_replica.cc:333] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:52.348340 2377 raft_consensus.cc:2243] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:52.348835 2377 raft_consensus.cc:2272] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:52.349704 2377 ts_tablet_manager.cc:1916] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:52.318179 4098 raft_consensus.cc:493] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:52.350011 4098 raft_consensus.cc:515] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.350821 4098 leader_election.cc:290] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:52.351198 2375 raft_consensus.cc:385] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.351342 2375 raft_consensus.cc:740] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.351706 2375 consensus_queue.cc:260] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:52.317970 2053 leader_election.cc:343] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.163551 1659 catalog_manager.cc:5741] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563) does not have the latest schema for tablet cfac07917ee34e6dba3fac5e081f6b43 (table default.Table_907991c2b0034c118065ead079f273f9 [id=a8d490f3dbc948d38c54d14e178b736e]). Expected version 1 got 0
W20251026 02:56:52.361570 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 223761us
I20251026 02:56:52.363986 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_591a2f198bfe45d1b90f60a732efc1e4" } new_table_name: "default.Table_fb31293471c24c16a70cc96d874e5443" modify_external_catalogs: true
W20251026 02:56:52.373981 2053 leader_election.cc:343] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.374379 2053 leader_election.cc:304] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:52.374759 4098 raft_consensus.cc:2749] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:52.380649 1968 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ded06ba1f80c4433ae78687e5990edce" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:52.382439 2375 ts_tablet_manager.cc:1434] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.234s user 0.002s sys 0.000s
I20251026 02:56:52.383312 2375 tablet_bootstrap.cc:492] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:52.385620 2375 tablet_bootstrap.cc:654] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:52.405133 1650 consensus_peers.cc:597] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7 -> Peer 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655): Couldn't send request to peer 647c0de4ce394caca75179fe11658fc8. Status: Network error: Client connection negotiation failed: client connection to 127.1.150.61:37655: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20251026 02:56:52.414016 1916 consensus_peers.cc:597] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b9a5daa36b547a7888d38dc3329eaf2. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:52.372841 2381 ts_tablet_manager.cc:1434] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.063s user 0.000s sys 0.002s
I20251026 02:56:52.415931 4069 raft_consensus.cc:493] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:52.416118 4069 raft_consensus.cc:515] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.417021 4069 leader_election.cc:290] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:52.417976 2381 tablet_bootstrap.cc:492] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:52.429268 2381 tablet_bootstrap.cc:654] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:52.432260 2053 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.1.150.60:33729, user_credentials={real_user=slave}} blocked reactor thread for 55565.9us
I20251026 02:56:52.436007 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aea01a248b345cf81d4d9862744eb78" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:52.436635 2110 raft_consensus.cc:2468] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
W20251026 02:56:52.443920 2053 leader_election.cc:343] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.435217 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ded06ba1f80c4433ae78687e5990edce" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:52.437151 4069 raft_consensus.cc:493] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:52.448014 4069 raft_consensus.cc:515] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.449051 4069 leader_election.cc:290] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:52.457338 4376 consensus_queue.cc:1048] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:52.464138 1968 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aea01a248b345cf81d4d9862744eb78" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:52.464568 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 9c7efb70c0804c149f36440a4a9e6ef8 (table default.Table_e99391c4c0f74eb3ac3b62cedada8612 [id=4397550364664316a6f1e96fb17d9509]) successfully deleted
W20251026 02:56:52.464814 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet abf58185766448d1a7c4f3d304b0c5b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abf58185766448d1a7c4f3d304b0c5b1 already in progress: creating tablet
W20251026 02:56:52.465018 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 163b8f785675474e90ff90a1fe6fc056 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 163b8f785675474e90ff90a1fe6fc056 already in progress: creating tablet
W20251026 02:56:52.465219 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet b2eab97dc2b9409697589a34e9cc7d6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2eab97dc2b9409697589a34e9cc7d6c already in progress: creating tablet
W20251026 02:56:52.465440 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 19016e8e11cc42c69ce44a819db9e298 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19016e8e11cc42c69ce44a819db9e298 already in progress: creating tablet
W20251026 02:56:52.475621 2194 leader_election.cc:343] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.480588 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ac38e6257e74367b06b15fc49d2af81" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:52.481227 2194 leader_election.cc:343] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.497805 2378 tablet_bootstrap.cc:492] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
W20251026 02:56:52.498356 1916 consensus_peers.cc:597] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ea364c4a5cf4f3f99d9ff6904add697. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:52.498704 2378 ts_tablet_manager.cc:1403] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.341s user 0.001s sys 0.005s
I20251026 02:56:52.499650 2378 raft_consensus.cc:359] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.501541 2378 raft_consensus.cc:385] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.502645 2378 raft_consensus.cc:740] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.502969 2378 consensus_queue.cc:260] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.509845 2378 ts_tablet_manager.cc:1434] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251026 02:56:52.510324 2378 tablet_bootstrap.cc:492] T 209fa71af13142a58efb9f3dab3eecd3 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:52.514019 2378 tablet_bootstrap.cc:654] T 209fa71af13142a58efb9f3dab3eecd3 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.516064 2372 ts_tablet_manager.cc:1929] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:52.516288 2372 log.cc:1199] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/bc4ca4423e3a4e0c96846babec931fec
I20251026 02:56:52.516844 2372 ts_tablet_manager.cc:1950] T bc4ca4423e3a4e0c96846babec931fec P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
W20251026 02:56:52.518317 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 55950us
I20251026 02:56:52.519572 2195 leader_election.cc:304] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:52.519857 4069 raft_consensus.cc:2804] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:52.519964 4069 raft_consensus.cc:493] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:52.520077 4069 raft_consensus.cc:3060] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:52.523003 4069 raft_consensus.cc:515] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.524179 4069 leader_election.cc:290] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:52.524430 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aea01a248b345cf81d4d9862744eb78" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:52.525156 2194 leader_election.cc:343] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.528975 2381 tablet_bootstrap.cc:492] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
W20251026 02:56:52.528739 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 61373.9us
I20251026 02:56:52.529212 2381 ts_tablet_manager.cc:1403] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.111s user 0.004s sys 0.000s
I20251026 02:56:52.536720 4098 raft_consensus.cc:493] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 696476265495492790739ec126a08539)
I20251026 02:56:52.536896 4098 raft_consensus.cc:515] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.537572 4098 leader_election.cc:290] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:52.542042 2381 raft_consensus.cc:359] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.542259 2381 raft_consensus.cc:385] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.542315 2381 raft_consensus.cc:740] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.542548 2381 consensus_queue.cc:260] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.543037 2381 ts_tablet_manager.cc:1434] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:52.543345 2381 tablet_bootstrap.cc:492] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:52.552978 2381 tablet_bootstrap.cc:654] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.554082 2375 tablet_bootstrap.cc:492] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:52.560724 2381 tablet_bootstrap.cc:492] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:52.560894 2381 ts_tablet_manager.cc:1403] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20251026 02:56:52.561650 2381 raft_consensus.cc:359] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.561829 2381 raft_consensus.cc:385] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.561878 2381 raft_consensus.cc:740] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.562063 2381 consensus_queue.cc:260] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.563498 2375 ts_tablet_manager.cc:1403] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.181s user 0.004s sys 0.000s
I20251026 02:56:52.563616 2381 ts_tablet_manager.cc:1434] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251026 02:56:52.564533 2381 tablet_bootstrap.cc:492] T 24f16e40a30c41788a6f9813c235c5ab P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:52.566414 2381 tablet_bootstrap.cc:654] T 24f16e40a30c41788a6f9813c235c5ab P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:52.566780 2194 consensus_peers.cc:597] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:52.573235 2376 ts_tablet_manager.cc:1929] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:52.581027 4440 raft_consensus.cc:493] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 209968db4dee47c49497a66c3ee750b7)
I20251026 02:56:52.581333 4440 raft_consensus.cc:515] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209968db4dee47c49497a66c3ee750b7" member_type: VOTER last_known_addr { host: "127.1.150.62" port: 44525 } } peers { permanent_uuid: "647c0de4ce394caca75179fe11658fc8" member_type: VOTER last_known_addr { host: "127.1.150.61" port: 37655 } } peers { permanent_uuid: "751b719b75ce4bc2a88a19091230ecb7" member_type: VOTER last_known_addr { host: "127.1.150.60" port: 33729 } }
I20251026 02:56:52.582712 1673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "751b719b75ce4bc2a88a19091230ecb7" candidate_term: 4 candidate_status { last_received { term: 3 index: 1569 } } ignore_live_leader: false dest_uuid: "209968db4dee47c49497a66c3ee750b7" is_pre_election: true
I20251026 02:56:52.582909 2381 tablet_bootstrap.cc:492] T 24f16e40a30c41788a6f9813c235c5ab P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:52.583030 2381 ts_tablet_manager.cc:1403] T 24f16e40a30c41788a6f9813c235c5ab P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.005s
I20251026 02:56:52.583703 2381 raft_consensus.cc:359] T 24f16e40a30c41788a6f9813c235c5ab P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.583886 2381 raft_consensus.cc:385] T 24f16e40a30c41788a6f9813c235c5ab P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.583941 2381 raft_consensus.cc:740] T 24f16e40a30c41788a6f9813c235c5ab P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.584151 2381 consensus_queue.cc:260] T 24f16e40a30c41788a6f9813c235c5ab P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.585736 2381 ts_tablet_manager.cc:1434] T 24f16e40a30c41788a6f9813c235c5ab P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 02:56:52.586719 2381 tablet_bootstrap.cc:492] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:52.587942 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ac38e6257e74367b06b15fc49d2af81" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:52.588155 2110 raft_consensus.cc:2468] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:52.589438 2381 tablet_bootstrap.cc:654] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.590233 4440 leader_election.cc:290] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 209968db4dee47c49497a66c3ee750b7 (127.1.150.62:44525), 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655)
W20251026 02:56:52.596268 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 551cc0916a21463d9190d14a31e79988 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 551cc0916a21463d9190d14a31e79988 already in progress: creating tablet
I20251026 02:56:52.597016 2375 raft_consensus.cc:359] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.597265 2375 raft_consensus.cc:385] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.597332 2375 raft_consensus.cc:740] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.597659 2381 tablet_bootstrap.cc:492] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:52.597556 2375 consensus_queue.cc:260] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.597815 2381 ts_tablet_manager.cc:1403] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20251026 02:56:52.598460 2381 raft_consensus.cc:359] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.598629 2381 raft_consensus.cc:385] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.598690 2381 raft_consensus.cc:740] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.598883 2381 consensus_queue.cc:260] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.600282 2381 ts_tablet_manager.cc:1434] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:52.601161 2381 tablet_bootstrap.cc:492] T db94738da3554fbda64c23fde3ef6c1b P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:52.602087 2053 leader_election.cc:343] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.602267 2053 leader_election.cc:304] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:52.596479 1647 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 130713us
W20251026 02:56:52.602806 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet fe27c85771964e4ca0ea747bbb4826df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe27c85771964e4ca0ea747bbb4826df already in progress: creating tablet
I20251026 02:56:52.602839 2110 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aea01a248b345cf81d4d9862744eb78" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:52.603060 2110 raft_consensus.cc:3060] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:52.603160 2381 tablet_bootstrap.cc:654] T db94738da3554fbda64c23fde3ef6c1b P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.607034 2376 log.cc:1199] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/99d318aa0d1847c58167bf0af9e75aa2
I20251026 02:56:52.608415 4437 tablet.cc:1722] T cfac07917ee34e6dba3fac5e081f6b43 P 0b6b0a6c4e144982b67980118cddba48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:52.609575 2376 ts_tablet_manager.cc:1950] T 99d318aa0d1847c58167bf0af9e75aa2 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:52.610662 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1088) took 3635 ms. Trace:
I20251026 02:56:52.610837 2376 rpcz_store.cc:276] 1026 02:56:48.975619 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.975688 (+ 69us) service_pool.cc:225] Handling call
1026 02:56:52.224367 (+3248679us) ts_tablet_manager.cc:1151] Deleting tablet 99d318aa0d1847c58167bf0af9e75aa2
1026 02:56:52.224376 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.371106 (+146730us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.573080 (+201974us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.610635 (+ 37555us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3248493}
I20251026 02:56:52.611039 2381 tablet_bootstrap.cc:492] T db94738da3554fbda64c23fde3ef6c1b P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:52.611150 2381 ts_tablet_manager.cc:1403] T db94738da3554fbda64c23fde3ef6c1b P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20251026 02:56:52.611877 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1090) took 3636 ms. Trace:
I20251026 02:56:52.612025 2376 rpcz_store.cc:276] 1026 02:56:48.975788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.975836 (+ 48us) service_pool.cc:225] Handling call
1026 02:56:52.611727 (+3635891us) ts_tablet_manager.cc:1151] Deleting tablet 8306f2ece961480d906df88f40b9740b
1026 02:56:52.611741 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.611859 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3635763}
I20251026 02:56:52.612094 2381 raft_consensus.cc:359] T db94738da3554fbda64c23fde3ef6c1b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.612267 2381 raft_consensus.cc:385] T db94738da3554fbda64c23fde3ef6c1b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.612319 2381 raft_consensus.cc:740] T db94738da3554fbda64c23fde3ef6c1b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.612521 2381 consensus_queue.cc:260] T db94738da3554fbda64c23fde3ef6c1b P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.613121 2381 ts_tablet_manager.cc:1434] T db94738da3554fbda64c23fde3ef6c1b P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:52.613361 2381 tablet_bootstrap.cc:492] T 81775055d98349f696cdab6171ffd58b P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:52.615059 2381 tablet_bootstrap.cc:654] T 81775055d98349f696cdab6171ffd58b P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.615087 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1089) took 3639 ms. Trace:
I20251026 02:56:52.615274 2376 rpcz_store.cc:276] 1026 02:56:48.975707 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.975905 (+ 198us) service_pool.cc:225] Handling call
1026 02:56:52.612671 (+3636766us) ts_tablet_manager.cc:1151] Deleting tablet 12e10fd463cd43c7abb22ef428c0aa37
1026 02:56:52.612765 (+ 94us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.615066 (+ 2301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3636655}
I20251026 02:56:52.615691 2376 tablet_replica.cc:333] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:52.618732 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 9b9a5daa36b547a7888d38dc3329eaf2 (table default.Table_217ad44ad25140129bf8e39c88b3b593 [id=26a5eae9586f4d00b5ebaf9eb3462272]) successfully deleted
W20251026 02:56:52.619005 1647 catalog_manager.cc:4952] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet ec13c5386abe414ba25589e03d5b850a because the tablet was not found. No further retry: Not found: Tablet not found: ec13c5386abe414ba25589e03d5b850a
I20251026 02:56:52.619431 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 99d318aa0d1847c58167bf0af9e75aa2 (table default.Table_62fd5a902d0648cea484f259d02de5f9 [id=3dc4d9941b7147adb84d1754e2783aa9]) successfully deleted
W20251026 02:56:52.619655 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 8306f2ece961480d906df88f40b9740b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8306f2ece961480d906df88f40b9740b already in progress: creating tablet
W20251026 02:56:52.619844 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet 12e10fd463cd43c7abb22ef428c0aa37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e10fd463cd43c7abb22ef428c0aa37 already in progress: creating tablet
I20251026 02:56:52.622140 2377 ts_tablet_manager.cc:1929] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:52.630534 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1085) took 5787 ms. Trace:
I20251026 02:56:52.630640 2372 rpcz_store.cc:276] 1026 02:56:46.842637 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.843368 (+ 731us) service_pool.cc:225] Handling call
1026 02:56:52.072779 (+5229411us) ts_tablet_manager.cc:1151] Deleting tablet bc4ca4423e3a4e0c96846babec931fec
1026 02:56:52.072791 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.390032 (+317241us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.515944 (+125912us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.630510 (+114566us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":111993,"tablet-delete.queue_time_us":5229157}
I20251026 02:56:52.631165 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet bc4ca4423e3a4e0c96846babec931fec (table default.Table_91f57fb05381498c89f99d19bdc7cd9b [id=05a30266e1a04e1490f58d1feb80b2a7]) successfully deleted
I20251026 02:56:52.632653 4098 raft_consensus.cc:2749] T ded06ba1f80c4433ae78687e5990edce P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:52.633214 4427 raft_consensus.cc:493] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:52.633337 4427 raft_consensus.cc:515] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.633812 4427 leader_election.cc:290] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:52.634594 2375 ts_tablet_manager.cc:1434] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.071s user 0.002s sys 0.000s
I20251026 02:56:52.639665 2375 tablet_bootstrap.cc:492] T f51f8fef342b46abafff5bf12dba7184 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:52.622782 2381 tablet_bootstrap.cc:492] T 81775055d98349f696cdab6171ffd58b P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:52.642424 2381 ts_tablet_manager.cc:1403] T 81775055d98349f696cdab6171ffd58b P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.004s
I20251026 02:56:52.643159 2381 raft_consensus.cc:359] T 81775055d98349f696cdab6171ffd58b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.643338 2381 raft_consensus.cc:385] T 81775055d98349f696cdab6171ffd58b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.643389 2381 raft_consensus.cc:740] T 81775055d98349f696cdab6171ffd58b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.643604 2381 consensus_queue.cc:260] T 81775055d98349f696cdab6171ffd58b P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.644227 2381 ts_tablet_manager.cc:1434] T 81775055d98349f696cdab6171ffd58b P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:52.644508 2381 tablet_bootstrap.cc:492] T c966cae6886048a2b6ad3b8bd7007bcb P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:52.646317 2381 tablet_bootstrap.cc:654] T c966cae6886048a2b6ad3b8bd7007bcb P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.634819 2372 tablet_replica.cc:333] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:52.648108 2372 raft_consensus.cc:2243] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:52.648146 2377 log.cc:1199] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/1eabfc95caa84afc9a8956a7ac13fea6
I20251026 02:56:52.648566 2377 ts_tablet_manager.cc:1950] T 1eabfc95caa84afc9a8956a7ac13fea6 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:52.648854 2372 raft_consensus.cc:2272] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:52.649331 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1174) took 3386 ms. Trace:
I20251026 02:56:52.649417 2377 rpcz_store.cc:276] 1026 02:56:49.262432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:49.262492 (+ 60us) service_pool.cc:225] Handling call
1026 02:56:52.347900 (+3085408us) ts_tablet_manager.cc:1151] Deleting tablet 1eabfc95caa84afc9a8956a7ac13fea6
1026 02:56:52.347918 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.525577 (+177659us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.622033 (+ 96456us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.649314 (+ 27281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3041796}
I20251026 02:56:52.649755 2377 tablet_replica.cc:333] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539: stopping tablet replica
W20251026 02:56:52.642845 2053 leader_election.cc:343] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: 3bcd7dec9c2149069e48dcf5e564595e
I20251026 02:56:52.649935 2377 raft_consensus.cc:2243] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:52.650090 2377 raft_consensus.cc:2272] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:52.656188 2381 tablet_bootstrap.cc:492] T c966cae6886048a2b6ad3b8bd7007bcb P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:52.659348 2381 ts_tablet_manager.cc:1403] T c966cae6886048a2b6ad3b8bd7007bcb P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20251026 02:56:52.659904 2377 ts_tablet_manager.cc:1916] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:52.660328 2381 raft_consensus.cc:359] T c966cae6886048a2b6ad3b8bd7007bcb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.660557 2381 raft_consensus.cc:385] T c966cae6886048a2b6ad3b8bd7007bcb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.660619 2381 raft_consensus.cc:740] T c966cae6886048a2b6ad3b8bd7007bcb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.660845 2381 consensus_queue.cc:260] T c966cae6886048a2b6ad3b8bd7007bcb P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.661453 2381 ts_tablet_manager.cc:1434] T c966cae6886048a2b6ad3b8bd7007bcb P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:52.661780 2381 tablet_bootstrap.cc:492] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:52.664038 2381 tablet_bootstrap.cc:654] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.669168 2381 tablet_bootstrap.cc:492] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:52.669353 2381 ts_tablet_manager.cc:1403] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20251026 02:56:52.669978 2381 raft_consensus.cc:359] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.670154 2381 raft_consensus.cc:385] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.670202 2381 raft_consensus.cc:740] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.670379 2381 consensus_queue.cc:260] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.670814 2381 ts_tablet_manager.cc:1434] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 02:56:52.671072 2381 tablet_bootstrap.cc:492] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:52.678042 2372 ts_tablet_manager.cc:1916] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:52.681913 2381 tablet_bootstrap.cc:654] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.685539 2377 ts_tablet_manager.cc:1929] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:52.687160 2110 raft_consensus.cc:2468] T 3aea01a248b345cf81d4d9862744eb78 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:52.689349 2053 leader_election.cc:343] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 3bcd7dec9c2149069e48dcf5e564595e
I20251026 02:56:52.691483 2053 leader_election.cc:304] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
W20251026 02:56:52.691735 2053 raft_consensus.cc:2686] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251026 02:56:52.694944 4428 raft_consensus.cc:493] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 02:56:52.695129 3337 leader_election.cc:336] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 647c0de4ce394caca75179fe11658fc8 (127.1.150.61:37655): Network error: Client connection negotiation failed: client connection to 127.1.150.61:37655: connect: Connection refused (error 111)
I20251026 02:56:52.695291 3337 leader_election.cc:304] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 751b719b75ce4bc2a88a19091230ecb7; no voters: 209968db4dee47c49497a66c3ee750b7, 647c0de4ce394caca75179fe11658fc8
I20251026 02:56:52.695173 4428 raft_consensus.cc:515] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.695534 4440 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 751b719b75ce4bc2a88a19091230ecb7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 02:56:52.695979 4428 leader_election.cc:290] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:52.695343 2381 tablet_bootstrap.cc:492] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:52.697755 2381 ts_tablet_manager.cc:1403] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20251026 02:56:52.698504 2381 raft_consensus.cc:359] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.698681 2381 raft_consensus.cc:385] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.698736 2381 raft_consensus.cc:740] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.698946 2381 consensus_queue.cc:260] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.699451 2381 ts_tablet_manager.cc:1434] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 02:56:52.699745 2381 tablet_bootstrap.cc:492] T 24449cb9121642009dec41220e27b193 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:52.701728 2381 tablet_bootstrap.cc:654] T 24449cb9121642009dec41220e27b193 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.701936 4387 raft_consensus.cc:493] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:52.702095 4387 raft_consensus.cc:515] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.702702 4387 leader_election.cc:290] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:52.704057 2195 leader_election.cc:304] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:52.706207 2377 log.cc:1199] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/f2d97084b5914350afbfda5c2a34bf9a
I20251026 02:56:52.707434 2377 ts_tablet_manager.cc:1950] T f2d97084b5914350afbfda5c2a34bf9a P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:52.708192 2375 tablet_bootstrap.cc:654] T f51f8fef342b46abafff5bf12dba7184 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.709215 4387 raft_consensus.cc:2804] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:52.709329 4387 raft_consensus.cc:493] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:52.709411 4387 raft_consensus.cc:3060] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:52.714073 4387 raft_consensus.cc:515] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.714704 4387 leader_election.cc:290] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:52.715818 4387 consensus_queue.cc:1048] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:52.717206 2195 consensus_peers.cc:597] T bc4ca4423e3a4e0c96846babec931fec P 696476265495492790739ec126a08539 -> Peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Couldn't send request to peer 0b6b0a6c4e144982b67980118cddba48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc4ca4423e3a4e0c96846babec931fec. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:52.719463 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eafb10b378e34ed3b4a33e56c6600759 consensus state has no leader
I20251026 02:56:52.720795 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1175) took 3458 ms. Trace:
I20251026 02:56:52.720896 2377 rpcz_store.cc:276] 1026 02:56:49.262567 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:49.262642 (+ 75us) service_pool.cc:225] Handling call
1026 02:56:52.649710 (+3387068us) ts_tablet_manager.cc:1151] Deleting tablet f2d97084b5914350afbfda5c2a34bf9a
1026 02:56:52.649719 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.674306 (+ 24587us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.685362 (+ 11056us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.720772 (+ 35410us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12434,"tablet-delete.queue_time_us":3340189}
I20251026 02:56:52.721439 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1177) took 3457 ms. Trace:
I20251026 02:56:52.721522 2377 rpcz_store.cc:276] 1026 02:56:49.263633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:49.263680 (+ 47us) service_pool.cc:225] Handling call
1026 02:56:52.721167 (+3457487us) ts_tablet_manager.cc:1151] Deleting tablet 122c81787d5c43d28b1fb639e3542c76
1026 02:56:52.721178 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.721421 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3411614}
I20251026 02:56:52.721810 2377 tablet_replica.cc:333] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:52.722009 2377 raft_consensus.cc:2243] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:52.722368 2377 raft_consensus.cc:2272] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:52.722959 2377 ts_tablet_manager.cc:1916] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:52.617040 2378 tablet_bootstrap.cc:492] T 209fa71af13142a58efb9f3dab3eecd3 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:52.724659 2378 ts_tablet_manager.cc:1403] T 209fa71af13142a58efb9f3dab3eecd3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.214s user 0.001s sys 0.002s
I20251026 02:56:52.725286 2378 raft_consensus.cc:359] T 209fa71af13142a58efb9f3dab3eecd3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.725435 2378 raft_consensus.cc:385] T 209fa71af13142a58efb9f3dab3eecd3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.725476 2378 raft_consensus.cc:740] T 209fa71af13142a58efb9f3dab3eecd3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.725680 2378 consensus_queue.cc:260] T 209fa71af13142a58efb9f3dab3eecd3 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.620045 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 87895fb88c284a4f8d7c58a5ee48ba8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:40 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:52.630115 4443 tablet.cc:1722] T cfac07917ee34e6dba3fac5e081f6b43 P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:52.632449 4378 consensus_queue.cc:1048] T 9fb42b54f558499090a1d7cf257dd84c P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:52.640514 4376 raft_consensus.cc:493] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:52.727473 4376 raft_consensus.cc:515] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.728127 4376 leader_election.cc:290] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:52.651290 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892a5fab2af1432eb31d92d9859ea2a0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:52.696691 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b49c0996c4d41cbaddcdf0b75846bcd" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:52.728855 2053 leader_election.cc:343] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:52.729182 2053 leader_election.cc:343] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.703441 1966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892a5fab2af1432eb31d92d9859ea2a0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:52.729626 2194 leader_election.cc:343] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.731403 4387 consensus_queue.cc:1048] T 66ede86c604e47dbb982e5a3ba18bcb7 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 02:56:52.734627 1916 leader_election.cc:343] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 87895fb88c284a4f8d7c58a5ee48ba8e
I20251026 02:56:52.736407 4376 raft_consensus.cc:493] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:52.736543 4376 raft_consensus.cc:515] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.737042 4376 leader_election.cc:290] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:52.738185 1916 leader_election.cc:343] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 41b7b0c36ba748f0a1565892dbe98a78
I20251026 02:56:52.739836 2195 leader_election.cc:304] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:52.740151 4387 raft_consensus.cc:2804] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 02:56:52.742388 2194 consensus_peers.cc:597] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:52.591954 1946 consensus_meta.cc:287] T 72e4a4ea953f4f2786a39c0ddbd11219 P 7fef01f4f2b843538ea245a09dd087a8: Time spent flushing consensus metadata: real 0.553s user 0.018s sys 0.001s
I20251026 02:56:52.742427 4387 raft_consensus.cc:697] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:52.742878 4387 consensus_queue.cc:237] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:52.708106 1915 consensus_peers.cc:597] T 9b9a5daa36b547a7888d38dc3329eaf2 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Couldn't send request to peer 696476265495492790739ec126a08539. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b9a5daa36b547a7888d38dc3329eaf2. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:52.715211 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ac38e6257e74367b06b15fc49d2af81" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:52.743556 2194 leader_election.cc:343] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.747130 2381 tablet_bootstrap.cc:492] T 24449cb9121642009dec41220e27b193 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:52.748718 2381 ts_tablet_manager.cc:1403] T 24449cb9121642009dec41220e27b193 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.002s
I20251026 02:56:52.749694 2381 raft_consensus.cc:359] T 24449cb9121642009dec41220e27b193 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.751466 2378 ts_tablet_manager.cc:1434] T 209fa71af13142a58efb9f3dab3eecd3 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.026s user 0.000s sys 0.008s
I20251026 02:56:52.751821 4428 raft_consensus.cc:493] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 696476265495492790739ec126a08539)
I20251026 02:56:52.751957 4428 raft_consensus.cc:515] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } }
I20251026 02:56:52.752532 4428 leader_election.cc:290] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:52.752847 2378 tablet_bootstrap.cc:492] T 85420a102dae4269811dbdc64aebb20a P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:52.754410 2378 tablet_bootstrap.cc:654] T 85420a102dae4269811dbdc64aebb20a P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:52.754869 2053 leader_election.cc:343] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Not found: Tablet not found: ce1ab21ddf8e45bc9063b63db9c1b97f
W20251026 02:56:52.756633 1649 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet eafb10b378e34ed3b4a33e56c6600759 (table default.Table_991a5e4bffdd4dc8b35db5027c582129 [id=71c92396f68f4ea480392c515e534360], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet eafb10b378e34ed3b4a33e56c6600759 consensus state has no leader
I20251026 02:56:52.764086 2381 raft_consensus.cc:385] T 24449cb9121642009dec41220e27b193 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.764336 2381 raft_consensus.cc:740] T 24449cb9121642009dec41220e27b193 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.764781 2381 consensus_queue.cc:260] T 24449cb9121642009dec41220e27b193 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.769342 1661 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_c8888ad957014cfd876933bf90eef89c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:52.769975 1661 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8888ad957014cfd876933bf90eef89c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:52.771099 4428 raft_consensus.cc:493] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:52.771229 4428 raft_consensus.cc:515] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.771840 4428 leader_election.cc:290] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:52.774019 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet 1eabfc95caa84afc9a8956a7ac13fea6 (table default.Table_0ea323ca655f45aa94b47eab713ebbd0 [id=1d82a5d03031489fa756df9cfb380b8e]) successfully deleted
I20251026 02:56:52.775058 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet f2d97084b5914350afbfda5c2a34bf9a (table default.Table_a7b1603151c54757bd2505c7958e3bae [id=cd7cfcd0eac044609fe5b2751ac6a98a]) successfully deleted
W20251026 02:56:52.775420 1647 catalog_manager.cc:4965] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): delete failed for tablet 122c81787d5c43d28b1fb639e3542c76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 122c81787d5c43d28b1fb639e3542c76 already in progress: creating tablet
I20251026 02:56:52.775887 1663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ba6ba94afd0e4f2ba4cb25e60f8cf508"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:52.776626 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892a5fab2af1432eb31d92d9859ea2a0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
W20251026 02:56:52.776779 1663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba6ba94afd0e4f2ba4cb25e60f8cf508 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:52.776904 2116 raft_consensus.cc:2468] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:52.777498 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ac38e6257e74367b06b15fc49d2af81" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:52.777622 2117 raft_consensus.cc:3060] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:52.778935 2381 ts_tablet_manager.cc:1434] T 24449cb9121642009dec41220e27b193 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20251026 02:56:52.779194 2381 tablet_bootstrap.cc:492] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539: Bootstrap starting.
W20251026 02:56:52.780361 1915 consensus_peers.cc:597] T c2d6691bb4cf4a8292b0c082e54a91a5 P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Couldn't send request to peer 696476265495492790739ec126a08539. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2d6691bb4cf4a8292b0c082e54a91a5. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:52.780802 1915 leader_election.cc:343] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 41b7b0c36ba748f0a1565892dbe98a78
I20251026 02:56:52.780892 2381 tablet_bootstrap.cc:654] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.780917 1915 leader_election.cc:304] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:52.777361 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87895fb88c284a4f8d7c58a5ee48ba8e" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:52.777827 2262 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892a5fab2af1432eb31d92d9859ea2a0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:52.777936 2260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b49c0996c4d41cbaddcdf0b75846bcd" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:52.781244 2262 raft_consensus.cc:2468] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:52.781440 2259 raft_consensus.cc:2468] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 0.
I20251026 02:56:52.783639 1966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28dfe285139042349b5b9d6cbc59a778" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:52.784859 1656 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_92904a5fabc348bb84e3369b454a4b0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:52.785411 1656 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92904a5fabc348bb84e3369b454a4b0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:52.785531 2053 leader_election.cc:343] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.788515 4460 consensus_queue.cc:1048] T 6ea364c4a5cf4f3f99d9ff6904add697 P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:52.789073 4460 raft_consensus.cc:2749] T 41b7b0c36ba748f0a1565892dbe98a78 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:52.792771 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b3bb056c94149bbb9f3dbfeb6db0d5a consensus state has no leader
I20251026 02:56:52.795678 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 3a55dc4eba9f41b089fe1266ecccba9a (DEFAULT_TABLE table=default.Table_2dbe045251c840efbf656f41a60ad6aa [id=4eb4a11d100b4be3bcb91ba7f2f6df83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.795820 2378 tablet_bootstrap.cc:492] T 85420a102dae4269811dbdc64aebb20a P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:52.795930 2378 ts_tablet_manager.cc:1403] T 85420a102dae4269811dbdc64aebb20a P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.002s
I20251026 02:56:52.796053 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a55dc4eba9f41b089fe1266ecccba9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.796625 2378 raft_consensus.cc:359] T 85420a102dae4269811dbdc64aebb20a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.796777 2378 raft_consensus.cc:385] T 85420a102dae4269811dbdc64aebb20a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.796819 2378 raft_consensus.cc:740] T 85420a102dae4269811dbdc64aebb20a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.796983 2378 consensus_queue.cc:260] T 85420a102dae4269811dbdc64aebb20a P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.797122 1658 catalog_manager.cc:5741] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611) does not have the latest schema for tablet cfac07917ee34e6dba3fac5e081f6b43 (table default.Table_907991c2b0034c118065ead079f273f9 [id=a8d490f3dbc948d38c54d14e178b736e]). Expected version 1 got 0
I20251026 02:56:52.799584 1657 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_ebfbe3fb80c64eeda4c4e24c93b0e139 [id=da72d631aee04d10a6832b04f208587f] alter complete (version 5)
I20251026 02:56:52.800165 2094 tablet_service.cc:1505] Processing CreateTablet for tablet fb4a644b60464637a2acf4b36e50e2c2 (DEFAULT_TABLE table=default.Table_85fc65eae0ab485abad03396cbe30fef [id=8dc29b993e6f4a6a80bb33c7e2c7d5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.800511 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4a644b60464637a2acf4b36e50e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:52.800886 1657 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.3:36919 (request call id 159) took 1067 ms (client timeout 1000 ms). Trace:
I20251026 02:56:52.801182 2378 ts_tablet_manager.cc:1434] T 85420a102dae4269811dbdc64aebb20a P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20251026 02:56:52.800961 1657 rpcz_store.cc:269] 1026 02:56:51.733625 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:51.742592 (+ 8967us) service_pool.cc:225] Handling call
1026 02:56:51.744663 (+ 2071us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x616000041254. stack: 00007f54638f59f6 00007f54638f57b0 00007f54628d13ad 0000000000983f04 0000000000983ecd 00007f5472bf36bd 00007f5472bf37b3 00007f546f7be441 00007f54774749dd 00007f54775fbaa8 00007f546e114ca6 00007f546e1149d5 00007f546a89127e 00007f546a89015e 00007f546a896395 00007f546a896cbf
1026 02:56:52.800864 (+1056201us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:51.939546 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:51.939626 (+ 80us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:51.939631 (+ 5us) write_op.cc:435] Acquired schema lock
1026 02:56:51.939633 (+ 2us) tablet.cc:660] Decoding operations
1026 02:56:51.939700 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:51.939706 (+ 6us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:51.939710 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1026 02:56:51.939746 (+ 36us) tablet.cc:699] Row locks acquired
1026 02:56:51.939748 (+ 2us) write_op.cc:260] PREPARE: finished
1026 02:56:51.939768 (+ 20us) write_op.cc:270] Start()
1026 02:56:51.939790 (+ 22us) write_op.cc:276] Timestamp: P: 1761447411939717 usec, L: 0
1026 02:56:51.939795 (+ 5us) op_driver.cc:348] REPLICATION: starting
1026 02:56:51.939909 (+ 114us) log.cc:844] Serialized 527 byte log entry
1026 02:56:51.985441 (+ 45532us) op_driver.cc:464] REPLICATION: finished
1026 02:56:51.985521 (+ 80us) write_op.cc:301] APPLY: starting
1026 02:56:51.985532 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:51.985558 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:51.985590 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:51.985629 (+ 39us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:51.985651 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:51.985658 (+ 7us) write_op.cc:312] APPLY: finished
1026 02:56:51.985727 (+ 69us) log.cc:844] Serialized 23 byte log entry
1026 02:56:51.985772 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:51.985784 (+ 12us) write_op.cc:454] Released schema lock
1026 02:56:51.985789 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1026 02:56:51.986784 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:51.986891 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:51.986894 (+ 3us) write_op.cc:435] Acquired schema lock
1026 02:56:51.986895 (+ 1us) tablet.cc:660] Decoding operations
1026 02:56:51.986967 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:51.986975 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:51.986977 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
1026 02:56:51.987022 (+ 45us) tablet.cc:699] Row locks acquired
1026 02:56:51.987023 (+ 1us) write_op.cc:260] PREPARE: finished
1026 02:56:51.987047 (+ 24us) write_op.cc:270] Start()
1026 02:56:51.987071 (+ 24us) write_op.cc:276] Timestamp: P: 1761447411986989 usec, L: 0
1026 02:56:51.987076 (+ 5us) op_driver.cc:348] REPLICATION: starting
1026 02:56:51.994442 (+ 7366us) log.cc:844] Serialized 356 byte log entry
1026 02:56:51.994598 (+ 156us) spinlock_profiling.cc:243] Waited 7.16 ms on lock 0x616000041254. stack: 00007f54638f59f6 00007f54638f57b0 00007f54628d13ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f546f790862 00007f5470b732a5 00007f5470b71dc9 00007f5470b7a1f0 00007f5470b79ffd 0000000000a8867c 00007f546393bd7f 00007f546393f04b 00007f546393eec8 0000000000a8867c
1026 02:56:52.016973 (+ 22375us) op_driver.cc:464] REPLICATION: finished
1026 02:56:52.017604 (+ 631us) write_op.cc:301] APPLY: starting
1026 02:56:52.017626 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:52.017673 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:52.017678 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:52.017747 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:52.017789 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 02:56:52.017800 (+ 11us) write_op.cc:312] APPLY: finished
1026 02:56:52.017889 (+ 89us) log.cc:844] Serialized 23 byte log entry
1026 02:56:52.017962 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:52.017984 (+ 22us) write_op.cc:454] Released schema lock
1026 02:56:52.017993 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:52.018077 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:52.018083 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":8,"spinlock_wait_cycles":2862848,"child_traces":[["op",{"apply.queue_time_us":44,"apply.run_cpu_time_us":314,"apply.run_wall_time_us":314,"num_ops":1,"prepare.queue_time_us":191194,"prepare.run_cpu_time_us":456,"prepare.run_wall_time_us":4749,"raft.queue_time_us":240,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":7,"replication_time_us":45602}],["op",{"apply.queue_time_us":558,"apply.run_cpu_time_us":466,"apply.run_wall_time_us":466,"num_ops":1,"prepare.queue_time_us":30,"prepare.run_cpu_time_us":828,"prepare.run_wall_time_us":7879,"raft.queue_time_us":2550,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":340,"replication_time_us":29852,"spinlock_wait_cycles":16470912}]]}
I20251026 02:56:52.802825 4098 raft_consensus.cc:493] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:52.802939 4098 raft_consensus.cc:515] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.803481 4098 leader_election.cc:290] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:52.804356 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1b6613742a24c21a1db6ebe858a846f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:52.805991 2378 tablet_bootstrap.cc:492] T 6699b50d87964f6a89c365171253ad25 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
W20251026 02:56:52.807036 1658 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.150.2:34337 (request call id 177) took 966 ms (client timeout 1000 ms). Trace:
W20251026 02:56:52.807122 1658 rpcz_store.cc:269] 1026 02:56:51.840531 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:51.894179 (+ 53648us) service_pool.cc:225] Handling call
1026 02:56:51.941215 (+ 47036us) spinlock_profiling.cc:243] Waited 45.7 ms on lock 0x616000041254. stack: 00007f54638f59f6 00007f54638f57b0 00007f54628d13ad 0000000000983f04 0000000000983ecd 00007f5472bf36bd 00007f5472bf37b3 00007f546f7be441 00007f54774749dd 00007f54775fbaa8 00007f546e114ca6 00007f546e1149d5 00007f546a89127e 00007f546a89015e 00007f546a896395 00007f546a896cbf
1026 02:56:52.807017 (+865802us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":105092352}
I20251026 02:56:52.807277 2378 tablet_bootstrap.cc:654] T 6699b50d87964f6a89c365171253ad25 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.808331 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet bc4ca4423e3a4e0c96846babec931fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:34 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:52.809366 2053 leader_election.cc:343] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.811098 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 3a55dc4eba9f41b089fe1266ecccba9a (DEFAULT_TABLE table=default.Table_2dbe045251c840efbf656f41a60ad6aa [id=4eb4a11d100b4be3bcb91ba7f2f6df83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.811434 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a55dc4eba9f41b089fe1266ecccba9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.813607 1961 tablet_service.cc:1505] Processing CreateTablet for tablet e76781e74d7d4e27bb93f39b78fb540e (DEFAULT_TABLE table=default.Table_5240cf82ceb146fb9fc20241d2cb5761 [id=cb055d1ab12043fea61d2b68f3f43ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.813855 2117 raft_consensus.cc:2468] T 3ac38e6257e74367b06b15fc49d2af81 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:52.813897 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e76781e74d7d4e27bb93f39b78fb540e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.816531 1962 tablet_service.cc:1505] Processing CreateTablet for tablet fb4a644b60464637a2acf4b36e50e2c2 (DEFAULT_TABLE table=default.Table_85fc65eae0ab485abad03396cbe30fef [id=8dc29b993e6f4a6a80bb33c7e2c7d5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.816864 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4a644b60464637a2acf4b36e50e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.819226 2086 tablet_service.cc:1505] Processing CreateTablet for tablet e76781e74d7d4e27bb93f39b78fb540e (DEFAULT_TABLE table=default.Table_5240cf82ceb146fb9fc20241d2cb5761 [id=cb055d1ab12043fea61d2b68f3f43ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.819609 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e76781e74d7d4e27bb93f39b78fb540e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.830279 1946 tablet_service.cc:1505] Processing CreateTablet for tablet 3d04802b9b304f33b9813c809605f065 (DEFAULT_TABLE table=default.Table_856686a17ddb4390908247e68a52742e [id=9ab93120136141db80aa5807edb9e7f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.831176 1946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d04802b9b304f33b9813c809605f065. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.831143 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 8423467856494a78acc7d5498b9f5506 (DEFAULT_TABLE table=default.Table_31f69698add24de2aaccc829dd116c38 [id=2a5d786e8ccf402bb45d6cfb1dfda2c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.831494 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8423467856494a78acc7d5498b9f5506. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.833683 4428 raft_consensus.cc:493] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:52.833820 4428 raft_consensus.cc:515] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.834441 4428 leader_election.cc:290] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:52.834559 1958 tablet_service.cc:1505] Processing CreateTablet for tablet 5aedb0d839b44ac887e769c593900f7b (DEFAULT_TABLE table=default.Table_09fa02d45d344077a3a72842956fa545 [id=fa64dcdecec0494dbc0a53d3853ab495]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.834846 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aedb0d839b44ac887e769c593900f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.835430 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 4143a74566e8440a9ead13544ac32129 (DEFAULT_TABLE table=default.Table_6c9225d34cdf4175bd75393736817b68 [id=3b8f157c02c841d2a80f182f1a2d03aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.835773 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4143a74566e8440a9ead13544ac32129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.836517 1947 tablet_service.cc:1505] Processing CreateTablet for tablet d81edc20febc45618393e9dc810d7c83 (DEFAULT_TABLE table=default.Table_2398d6b14bee449e97556f1c169d3f77 [id=8efadba4fea6497189435297ee2dca5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.836804 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d81edc20febc45618393e9dc810d7c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.838099 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81775055d98349f696cdab6171ffd58b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:52.840272 2053 leader_election.cc:343] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.840463 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 67cb4a4a87a44304a132cc0fc0bf7bc3 (DEFAULT_TABLE table=default.Table_cb422e5dd84d4d50ac55af4385a3b06b [id=a7e2e484f15649aea6243dab8f2b7c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.840749 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67cb4a4a87a44304a132cc0fc0bf7bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.844506 1959 tablet_service.cc:1505] Processing CreateTablet for tablet 073b67cccd6b47fdbd64917267c0b4dd (DEFAULT_TABLE table=default.Table_fce269332a224a298ef59f900f8c8d03 [id=79afe4f271ec4691be45a5b20d422176]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.844825 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073b67cccd6b47fdbd64917267c0b4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.848505 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 6d30f806bf2a4c3685d8398be78e91d9 (DEFAULT_TABLE table=default.Table_e88975df66e940a89e6353445cc36c7e [id=494dc3b5dda24b5eb68d8d5ba1da7552]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.848840 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d30f806bf2a4c3685d8398be78e91d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.852576 1944 tablet_service.cc:1505] Processing CreateTablet for tablet efac562cb34549d4bf849b6cdd4ae20a (DEFAULT_TABLE table=default.Table_eb26fd925b0748debe01dd229ed39094 [id=ad485ef100a64c4ea0aecf55c4e14afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.852991 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efac562cb34549d4bf849b6cdd4ae20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.867029 4428 raft_consensus.cc:493] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:52.867249 4428 raft_consensus.cc:515] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.868111 4428 leader_election.cc:290] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:52.869069 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09f5265d85c6496d85c555b2b847373a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:52.869851 2053 leader_election.cc:343] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:52.894527 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet e830f1aa1270420c9b28a0491a1a0960 with delete_type TABLET_DATA_DELETED (replaced by tablet 8423becb5b214f32a66445cd351b200f) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:52.895076 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 66c0d1b731074ee1acc9c3032c8bef71 (DEFAULT_TABLE table=default.Table_cd4901edb6854b86b0076a24935350ea [id=433c6052152945b6946654fdc937a361]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.895537 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c0d1b731074ee1acc9c3032c8bef71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.896605 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 66c0d1b731074ee1acc9c3032c8bef71 (DEFAULT_TABLE table=default.Table_cd4901edb6854b86b0076a24935350ea [id=433c6052152945b6946654fdc937a361]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.897239 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c0d1b731074ee1acc9c3032c8bef71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.897989 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 3d04802b9b304f33b9813c809605f065 (DEFAULT_TABLE table=default.Table_856686a17ddb4390908247e68a52742e [id=9ab93120136141db80aa5807edb9e7f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.898356 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d04802b9b304f33b9813c809605f065. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.898543 2085 tablet_service.cc:1505] Processing CreateTablet for tablet 67cb4a4a87a44304a132cc0fc0bf7bc3 (DEFAULT_TABLE table=default.Table_cb422e5dd84d4d50ac55af4385a3b06b [id=a7e2e484f15649aea6243dab8f2b7c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.898828 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67cb4a4a87a44304a132cc0fc0bf7bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.901192 2088 tablet_service.cc:1505] Processing CreateTablet for tablet d81edc20febc45618393e9dc810d7c83 (DEFAULT_TABLE table=default.Table_2398d6b14bee449e97556f1c169d3f77 [id=8efadba4fea6497189435297ee2dca5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.901621 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d81edc20febc45618393e9dc810d7c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.903324 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 8423467856494a78acc7d5498b9f5506 (DEFAULT_TABLE table=default.Table_31f69698add24de2aaccc829dd116c38 [id=2a5d786e8ccf402bb45d6cfb1dfda2c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.903674 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8423467856494a78acc7d5498b9f5506. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.904006 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 073b67cccd6b47fdbd64917267c0b4dd (DEFAULT_TABLE table=default.Table_fce269332a224a298ef59f900f8c8d03 [id=79afe4f271ec4691be45a5b20d422176]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.904314 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073b67cccd6b47fdbd64917267c0b4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.917008 2091 tablet_service.cc:1505] Processing CreateTablet for tablet 5aedb0d839b44ac887e769c593900f7b (DEFAULT_TABLE table=default.Table_09fa02d45d344077a3a72842956fa545 [id=fa64dcdecec0494dbc0a53d3853ab495]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.917626 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aedb0d839b44ac887e769c593900f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.920320 2376 raft_consensus.cc:2243] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:52.920506 2376 raft_consensus.cc:2272] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:52.920624 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 4143a74566e8440a9ead13544ac32129 (DEFAULT_TABLE table=default.Table_6c9225d34cdf4175bd75393736817b68 [id=3b8f157c02c841d2a80f182f1a2d03aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.921036 2376 ts_tablet_manager.cc:1916] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:52.921084 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4143a74566e8440a9ead13544ac32129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.924674 2100 tablet_service.cc:1505] Processing CreateTablet for tablet efac562cb34549d4bf849b6cdd4ae20a (DEFAULT_TABLE table=default.Table_eb26fd925b0748debe01dd229ed39094 [id=ad485ef100a64c4ea0aecf55c4e14afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.927802 2381 tablet_bootstrap.cc:492] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:52.927948 2381 ts_tablet_manager.cc:1403] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.149s user 0.003s sys 0.000s
I20251026 02:56:52.928670 2381 raft_consensus.cc:359] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.928836 2381 raft_consensus.cc:385] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.928881 2381 raft_consensus.cc:740] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:52.929057 2381 consensus_queue.cc:260] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.929472 2381 ts_tablet_manager.cc:1434] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 02:56:52.929780 2381 tablet_bootstrap.cc:492] T b1b6613742a24c21a1db6ebe858a846f P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:52.931365 2381 tablet_bootstrap.cc:654] T b1b6613742a24c21a1db6ebe858a846f P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.937357 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efac562cb34549d4bf849b6cdd4ae20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:52.939483 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 6d30f806bf2a4c3685d8398be78e91d9 (DEFAULT_TABLE table=default.Table_e88975df66e940a89e6353445cc36c7e [id=494dc3b5dda24b5eb68d8d5ba1da7552]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:52.939867 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d30f806bf2a4c3685d8398be78e91d9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:52.945511 2194 consensus_peers.cc:597] T a3af2695c41a4439bc4092b7cb1b2edf P 696476265495492790739ec126a08539 -> Peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Couldn't send request to peer 7fef01f4f2b843538ea245a09dd087a8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 02:56:52.951151 2372 ts_tablet_manager.cc:1929] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 02:56:52.952461 1981 raft_consensus.cc:3060] T 209fa71af13142a58efb9f3dab3eecd3 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:52.969951 2381 tablet_bootstrap.cc:492] T b1b6613742a24c21a1db6ebe858a846f P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:52.970818 2381 ts_tablet_manager.cc:1403] T b1b6613742a24c21a1db6ebe858a846f P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.005s
I20251026 02:56:52.976327 4460 raft_consensus.cc:493] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 696476265495492790739ec126a08539)
I20251026 02:56:52.976612 4460 raft_consensus.cc:515] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.971859 2381 raft_consensus.cc:359] T b1b6613742a24c21a1db6ebe858a846f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.978032 2381 raft_consensus.cc:385] T b1b6613742a24c21a1db6ebe858a846f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:52.978293 2381 raft_consensus.cc:740] T b1b6613742a24c21a1db6ebe858a846f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
W20251026 02:56:52.978504 1916 leader_election.cc:343] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 4a3f3fc3c7a04d05bae28472251f3acc
I20251026 02:56:52.978718 4460 leader_election.cc:290] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:52.978863 2381 consensus_queue.cc:260] T b1b6613742a24c21a1db6ebe858a846f P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.979423 2381 ts_tablet_manager.cc:1434] T b1b6613742a24c21a1db6ebe858a846f P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251026 02:56:52.979676 2381 tablet_bootstrap.cc:492] T 09f5265d85c6496d85c555b2b847373a P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:52.982656 2381 tablet_bootstrap.cc:654] T 09f5265d85c6496d85c555b2b847373a P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:52.985559 4460 raft_consensus.cc:493] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:52.985774 4460 raft_consensus.cc:515] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:52.986677 4460 leader_election.cc:290] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:52.987303 1916 leader_election.cc:343] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: dfd3e23d83a84648bd11cd258377aecb
I20251026 02:56:52.987455 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_0ec238d9794d43ed9faa893c9becaddb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:52.987818 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_4130d1886ff34713a778ea2b3f872444"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:52.988085 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ec238d9794d43ed9faa893c9becaddb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:52.988269 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4130d1886ff34713a778ea2b3f872444 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:52.998546 1658 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_21830ac7f7ef4cfdb68626b890c460b6" } modify_external_catalogs: true
I20251026 02:56:52.998759 1658 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_21830ac7f7ef4cfdb68626b890c460b6" } modify_external_catalogs: true
I20251026 02:56:53.005957 2376 ts_tablet_manager.cc:1929] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:53.006095 2376 log.cc:1199] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/1fb2ff87cf0248a0a4a77f2733ef867b
I20251026 02:56:53.006474 2376 ts_tablet_manager.cc:1950] T 1fb2ff87cf0248a0a4a77f2733ef867b P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:53.002627 2195 leader_election.cc:304] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.016892 4255 raft_consensus.cc:2804] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:53.016988 4255 raft_consensus.cc:493] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:53.017048 4255 raft_consensus.cc:3060] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.017258 2195 leader_election.cc:304] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.017585 4069 raft_consensus.cc:2804] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:53.016147 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f65b2167c56c419b9b5fd835ce39e3ed" } new_table_name: "default.Table_3be2b652dee04ce3bd96f36d4d2eae2e" modify_external_catalogs: true
I20251026 02:56:53.019745 4255 raft_consensus.cc:515] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.021674 4255 leader_election.cc:290] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:53.023672 1662 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 283939e86520421b839f9c1af0ddeba3
I20251026 02:56:53.024439 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 89a7853ff92c4350822b205a322fc62b
I20251026 02:56:53.024964 1654 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39910 (request call id 1506) took 1063 ms. Trace:
I20251026 02:56:53.025065 1654 rpcz_store.cc:276] 1026 02:56:51.961296 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:52.021184 (+ 59888us) service_pool.cc:225] Handling call
1026 02:56:52.034796 (+ 13612us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x616000041254. stack: 00007f54638f59f6 00007f54638f57b0 00007f54628d13ad 0000000000983f04 0000000000983ecd 00007f5472bf36bd 00007f5472bf37b3 00007f546f7be441 00007f54774749dd 00007f5477602256 00007f546e119486 00007f546e1191b5 00007f546a89127e 00007f546a89015e 00007f546a896395 00007f546a896cbf
1026 02:56:52.035208 (+ 412us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1026 02:56:52.035374 (+ 166us) catalog_manager.cc:2815] Modifying in-memory table state
1026 02:56:52.035461 (+ 87us) catalog_manager.cc:2824] Locking tablets
1026 02:56:52.799079 (+763618us) catalog_manager.cc:2838] Removing table and tablets from system table
1026 02:56:53.024289 (+225210us) catalog_manager.cc:2856] Removing table from by-name map
1026 02:56:53.024373 (+ 84us) catalog_manager.cc:2872] Committing in-memory state
1026 02:56:53.024413 (+ 40us) catalog_manager.cc:2876] Aborting table tasks
1026 02:56:53.024942 (+ 529us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:53.000957 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:53.001040 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:53.001044 (+ 4us) write_op.cc:435] Acquired schema lock
1026 02:56:53.001046 (+ 2us) tablet.cc:660] Decoding operations
1026 02:56:53.001128 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:53.001136 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:53.001140 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1026 02:56:53.001212 (+ 72us) tablet.cc:699] Row locks acquired
1026 02:56:53.001214 (+ 2us) write_op.cc:260] PREPARE: finished
1026 02:56:53.001238 (+ 24us) write_op.cc:270] Start()
1026 02:56:53.001262 (+ 24us) write_op.cc:276] Timestamp: P: 1761447413001185 usec, L: 0
1026 02:56:53.001265 (+ 3us) op_driver.cc:348] REPLICATION: starting
1026 02:56:53.001389 (+ 124us) log.cc:844] Serialized 725 byte log entry
1026 02:56:53.020474 (+ 19085us) op_driver.cc:464] REPLICATION: finished
1026 02:56:53.022450 (+ 1976us) write_op.cc:301] APPLY: starting
1026 02:56:53.022459 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:53.022485 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:53.022517 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:53.022589 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:53.022642 (+ 53us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1026 02:56:53.022651 (+ 9us) write_op.cc:312] APPLY: finished
1026 02:56:53.022711 (+ 60us) log.cc:844] Serialized 29 byte log entry
1026 02:56:53.022761 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:53.022773 (+ 12us) write_op.cc:454] Released schema lock
1026 02:56:53.022778 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:53.022864 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:53.022869 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":30501120,"child_traces":[["op",{"apply.queue_time_us":1943,"apply.run_cpu_time_us":404,"apply.run_wall_time_us":404,"num_ops":2,"prepare.queue_time_us":201434,"prepare.run_cpu_time_us":485,"prepare.run_wall_time_us":484,"replication_time_us":19190}]]}
I20251026 02:56:53.027127 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 09f5265d85c6496d85c555b2b847373a
I20251026 02:56:53.027629 1660 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39910 (request call id 1501) took 1141 ms. Trace:
I20251026 02:56:53.027700 1660 rpcz_store.cc:276] 1026 02:56:51.885743 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:51.984527 (+ 98784us) service_pool.cc:225] Handling call
1026 02:56:51.994357 (+ 9830us) spinlock_profiling.cc:243] Waited 9.63 ms on lock 0x616000041254. stack: 00007f54638f59f6 00007f54638f57b0 00007f54628d13ad 0000000000983f04 0000000000983ecd 00007f5472bf36bd 00007f5472bf37b3 00007f546f7be441 00007f54774749dd 00007f5477602256 00007f546e119486 00007f546e1191b5 00007f546a89127e 00007f546a89015e 00007f546a896395 00007f546a896cbf
1026 02:56:51.994592 (+ 235us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1026 02:56:51.994707 (+ 115us) catalog_manager.cc:2815] Modifying in-memory table state
1026 02:56:51.994753 (+ 46us) catalog_manager.cc:2824] Locking tablets
1026 02:56:52.797634 (+802881us) catalog_manager.cc:2838] Removing table and tablets from system table
1026 02:56:53.023348 (+225714us) catalog_manager.cc:2856] Removing table from by-name map
1026 02:56:53.023446 (+ 98us) catalog_manager.cc:2872] Committing in-memory state
1026 02:56:53.023491 (+ 45us) catalog_manager.cc:2876] Aborting table tasks
1026 02:56:53.027614 (+ 4123us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:52.999810 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:52.999916 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:52.999920 (+ 4us) write_op.cc:435] Acquired schema lock
1026 02:56:52.999922 (+ 2us) tablet.cc:660] Decoding operations
1026 02:56:53.000018 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:53.000025 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:53.000029 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1026 02:56:53.000110 (+ 81us) tablet.cc:699] Row locks acquired
1026 02:56:53.000112 (+ 2us) write_op.cc:260] PREPARE: finished
1026 02:56:53.000137 (+ 25us) write_op.cc:270] Start()
1026 02:56:53.000162 (+ 25us) write_op.cc:276] Timestamp: P: 1761447413000085 usec, L: 0
1026 02:56:53.000166 (+ 4us) op_driver.cc:348] REPLICATION: starting
1026 02:56:53.000324 (+ 158us) log.cc:844] Serialized 756 byte log entry
1026 02:56:53.020359 (+ 20035us) op_driver.cc:464] REPLICATION: finished
1026 02:56:53.021351 (+ 992us) write_op.cc:301] APPLY: starting
1026 02:56:53.021376 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:53.021437 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:53.021469 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:53.021576 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:53.021610 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1026 02:56:53.021617 (+ 7us) write_op.cc:312] APPLY: finished
1026 02:56:53.021694 (+ 77us) log.cc:844] Serialized 29 byte log entry
1026 02:56:53.021787 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:53.021811 (+ 24us) write_op.cc:454] Released schema lock
1026 02:56:53.021819 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:53.021896 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:53.021903 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3585,"spinlock_wait_cycles":22160128,"child_traces":[["op",{"apply.queue_time_us":917,"apply.run_cpu_time_us":544,"apply.run_wall_time_us":545,"num_ops":2,"prepare.queue_time_us":201542,"prepare.run_cpu_time_us":579,"prepare.run_wall_time_us":579,"replication_time_us":20164}]]}
I20251026 02:56:53.030299 1662 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39910 (request call id 1499) took 1186 ms. Trace:
I20251026 02:56:53.030385 1662 rpcz_store.cc:276] 1026 02:56:51.844173 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:51.981053 (+136880us) service_pool.cc:225] Handling call
1026 02:56:51.993658 (+ 12605us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x616000041254. stack: 00007f54638f59f6 00007f54638f57b0 00007f54628d13ad 0000000000983f04 0000000000983ecd 00007f5472bf36bd 00007f5472bf37b3 00007f546f7be441 00007f54774749dd 00007f5477602256 00007f546e119486 00007f546e1191b5 00007f546a89127e 00007f546a89015e 00007f546a896395 00007f546a896cbf
1026 02:56:51.993949 (+ 291us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1026 02:56:51.994124 (+ 175us) catalog_manager.cc:2815] Modifying in-memory table state
1026 02:56:51.994183 (+ 59us) catalog_manager.cc:2824] Locking tablets
1026 02:56:52.798427 (+804244us) catalog_manager.cc:2838] Removing table and tablets from system table
1026 02:56:53.023559 (+225132us) catalog_manager.cc:2856] Removing table from by-name map
1026 02:56:53.023618 (+ 59us) catalog_manager.cc:2872] Committing in-memory state
1026 02:56:53.023643 (+ 25us) catalog_manager.cc:2876] Aborting table tasks
1026 02:56:53.030283 (+ 6640us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 02:56:53.000416 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 02:56:53.000503 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
1026 02:56:53.000507 (+ 4us) write_op.cc:435] Acquired schema lock
1026 02:56:53.000520 (+ 13us) tablet.cc:660] Decoding operations
1026 02:56:53.000607 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
1026 02:56:53.000615 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1026 02:56:53.000619 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1026 02:56:53.000711 (+ 92us) tablet.cc:699] Row locks acquired
1026 02:56:53.000713 (+ 2us) write_op.cc:260] PREPARE: finished
1026 02:56:53.000736 (+ 23us) write_op.cc:270] Start()
1026 02:56:53.000758 (+ 22us) write_op.cc:276] Timestamp: P: 1761447413000685 usec, L: 0
1026 02:56:53.000761 (+ 3us) op_driver.cc:348] REPLICATION: starting
1026 02:56:53.000872 (+ 111us) log.cc:844] Serialized 725 byte log entry
1026 02:56:53.020429 (+ 19557us) op_driver.cc:464] REPLICATION: finished
1026 02:56:53.021983 (+ 1554us) write_op.cc:301] APPLY: starting
1026 02:56:53.021994 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1026 02:56:53.022024 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
1026 02:56:53.022043 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 02:56:53.022092 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 02:56:53.022120 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1026 02:56:53.022128 (+ 8us) write_op.cc:312] APPLY: finished
1026 02:56:53.022195 (+ 67us) log.cc:844] Serialized 29 byte log entry
1026 02:56:53.022272 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:53.022289 (+ 17us) write_op.cc:454] Released schema lock
1026 02:56:53.022297 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1026 02:56:53.022376 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
1026 02:56:53.022382 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":28458752,"child_traces":[["op",{"apply.queue_time_us":1516,"apply.run_cpu_time_us":381,"apply.run_wall_time_us":381,"num_ops":2,"prepare.queue_time_us":201502,"prepare.run_cpu_time_us":513,"prepare.run_wall_time_us":512,"replication_time_us":19650}]]}
I20251026 02:56:52.955686 4428 raft_consensus.cc:493] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:53.072594 2381 tablet_bootstrap.cc:492] T 09f5265d85c6496d85c555b2b847373a P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:53.072458 4428 raft_consensus.cc:515] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.072754 2381 ts_tablet_manager.cc:1403] T 09f5265d85c6496d85c555b2b847373a P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.093s user 0.010s sys 0.009s
I20251026 02:56:53.073289 4428 leader_election.cc:290] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:52.999605 2375 tablet_bootstrap.cc:492] T f51f8fef342b46abafff5bf12dba7184 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:53.073546 2375 ts_tablet_manager.cc:1403] T f51f8fef342b46abafff5bf12dba7184 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.434s user 0.001s sys 0.006s
I20251026 02:56:53.073673 2381 raft_consensus.cc:359] T 09f5265d85c6496d85c555b2b847373a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.073873 2381 raft_consensus.cc:385] T 09f5265d85c6496d85c555b2b847373a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:53.073930 2381 raft_consensus.cc:740] T 09f5265d85c6496d85c555b2b847373a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:53.074208 2381 consensus_queue.cc:260] T 09f5265d85c6496d85c555b2b847373a P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.074425 2375 raft_consensus.cc:359] T f51f8fef342b46abafff5bf12dba7184 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.074620 2375 raft_consensus.cc:385] T f51f8fef342b46abafff5bf12dba7184 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:53.074671 2375 raft_consensus.cc:740] T f51f8fef342b46abafff5bf12dba7184 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:53.074857 2375 consensus_queue.cc:260] T f51f8fef342b46abafff5bf12dba7184 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.075366 2375 ts_tablet_manager.cc:1434] T f51f8fef342b46abafff5bf12dba7184 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 02:56:53.075704 2375 tablet_bootstrap.cc:492] T 1a1b8c440061429caea0f0803f110742 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
W20251026 02:56:53.017863 2053 leader_election.cc:343] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.027072 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 283939e86520421b839f9c1af0ddeba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:53.027243 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 89a7853ff92c4350822b205a322fc62b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:52 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:53.076771 2053 leader_election.cc:304] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.029173 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 09f5265d85c6496d85c555b2b847373a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:53.077206 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 59352.7us
I20251026 02:56:53.077785 2053 leader_election.cc:304] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.078153 4427 raft_consensus.cc:2749] T 4b49c0996c4d41cbaddcdf0b75846bcd P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 02:56:53.078240 2053 leader_election.cc:343] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: ce1ab21ddf8e45bc9063b63db9c1b97f
I20251026 02:56:53.078341 4427 raft_consensus.cc:2804] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:53.078327 2053 leader_election.cc:304] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.078397 4427 raft_consensus.cc:493] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:53.078450 4427 raft_consensus.cc:3060] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.079854 2241 tablet_service.cc:1505] Processing CreateTablet for tablet fb4a644b60464637a2acf4b36e50e2c2 (DEFAULT_TABLE table=default.Table_85fc65eae0ab485abad03396cbe30fef [id=8dc29b993e6f4a6a80bb33c7e2c7d5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.080302 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4a644b60464637a2acf4b36e50e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.080457 4098 raft_consensus.cc:2749] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:53.080969 2230 tablet_service.cc:1505] Processing CreateTablet for tablet 3a55dc4eba9f41b089fe1266ecccba9a (DEFAULT_TABLE table=default.Table_2dbe045251c840efbf656f41a60ad6aa [id=4eb4a11d100b4be3bcb91ba7f2f6df83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.081264 2228 tablet_service.cc:1505] Processing CreateTablet for tablet 5aedb0d839b44ac887e769c593900f7b (DEFAULT_TABLE table=default.Table_09fa02d45d344077a3a72842956fa545 [id=fa64dcdecec0494dbc0a53d3853ab495]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.081449 2229 tablet_service.cc:1505] Processing CreateTablet for tablet e76781e74d7d4e27bb93f39b78fb540e (DEFAULT_TABLE table=default.Table_5240cf82ceb146fb9fc20241d2cb5761 [id=cb055d1ab12043fea61d2b68f3f43ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.081851 2239 tablet_service.cc:1505] Processing CreateTablet for tablet 8423467856494a78acc7d5498b9f5506 (DEFAULT_TABLE table=default.Table_31f69698add24de2aaccc829dd116c38 [id=2a5d786e8ccf402bb45d6cfb1dfda2c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.081960 2234 tablet_service.cc:1505] Processing CreateTablet for tablet 073b67cccd6b47fdbd64917267c0b4dd (DEFAULT_TABLE table=default.Table_fce269332a224a298ef59f900f8c8d03 [id=79afe4f271ec4691be45a5b20d422176]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.082247 2238 tablet_service.cc:1505] Processing CreateTablet for tablet efac562cb34549d4bf849b6cdd4ae20a (DEFAULT_TABLE table=default.Table_eb26fd925b0748debe01dd229ed39094 [id=ad485ef100a64c4ea0aecf55c4e14afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.082515 2227 tablet_service.cc:1505] Processing CreateTablet for tablet 4143a74566e8440a9ead13544ac32129 (DEFAULT_TABLE table=default.Table_6c9225d34cdf4175bd75393736817b68 [id=3b8f157c02c841d2a80f182f1a2d03aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.082592 2226 tablet_service.cc:1505] Processing CreateTablet for tablet 6d30f806bf2a4c3685d8398be78e91d9 (DEFAULT_TABLE table=default.Table_e88975df66e940a89e6353445cc36c7e [id=494dc3b5dda24b5eb68d8d5ba1da7552]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.080983 4427 raft_consensus.cc:515] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.083709 2237 tablet_service.cc:1552] Processing DeleteTablet for tablet 09f5265d85c6496d85c555b2b847373a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.083881 2235 tablet_service.cc:1552] Processing DeleteTablet for tablet b8fe7825fa8d46669772cf84bc4b3afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:41 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.082223 2231 tablet_service.cc:1505] Processing CreateTablet for tablet 67cb4a4a87a44304a132cc0fc0bf7bc3 (DEFAULT_TABLE table=default.Table_cb422e5dd84d4d50ac55af4385a3b06b [id=a7e2e484f15649aea6243dab8f2b7c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.084031 2233 tablet_service.cc:1552] Processing DeleteTablet for tablet a3af2695c41a4439bc4092b7cb1b2edf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:31 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.084137 2240 tablet_service.cc:1552] Processing DeleteTablet for tablet dfd3e23d83a84648bd11cd258377aecb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.084434 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 66c0d1b731074ee1acc9c3032c8bef71 (DEFAULT_TABLE table=default.Table_cd4901edb6854b86b0076a24935350ea [id=433c6052152945b6946654fdc937a361]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.084625 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28dfe285139042349b5b9d6cbc59a778" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:53.084754 2224 tablet_service.cc:1505] Processing CreateTablet for tablet d81edc20febc45618393e9dc810d7c83 (DEFAULT_TABLE table=default.Table_2398d6b14bee449e97556f1c169d3f77 [id=8efadba4fea6497189435297ee2dca5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.085040 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1b6613742a24c21a1db6ebe858a846f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:53.085186 2255 raft_consensus.cc:2468] T b1b6613742a24c21a1db6ebe858a846f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:53.085376 2243 tablet_service.cc:1552] Processing DeleteTablet for tablet 283939e86520421b839f9c1af0ddeba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.085580 2225 tablet_service.cc:1552] Processing DeleteTablet for tablet 89a7853ff92c4350822b205a322fc62b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:52 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.080307 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 3d04802b9b304f33b9813c809605f065 (DEFAULT_TABLE table=default.Table_856686a17ddb4390908247e68a52742e [id=9ab93120136141db80aa5807edb9e7f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.081276 2230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a55dc4eba9f41b089fe1266ecccba9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.086246 2375 tablet_bootstrap.cc:654] T 1a1b8c440061429caea0f0803f110742 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:53.086300 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d04802b9b304f33b9813c809605f065. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.086546 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e76781e74d7d4e27bb93f39b78fb540e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.089615 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efac562cb34549d4bf849b6cdd4ae20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.092394 2255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81775055d98349f696cdab6171ffd58b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:53.092563 2255 raft_consensus.cc:2468] T 81775055d98349f696cdab6171ffd58b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:53.092796 2263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24f16e40a30c41788a6f9813c235c5ab" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:53.092922 2263 raft_consensus.cc:2468] T 24f16e40a30c41788a6f9813c235c5ab P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:53.093086 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09f5265d85c6496d85c555b2b847373a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:53.093194 2256 raft_consensus.cc:2468] T 09f5265d85c6496d85c555b2b847373a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:53.093290 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892a5fab2af1432eb31d92d9859ea2a0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:53.093387 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4143a74566e8440a9ead13544ac32129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.093534 2118 raft_consensus.cc:2393] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 696476265495492790739ec126a08539 in current term 1: Already voted for candidate 0b6b0a6c4e144982b67980118cddba48 in this term.
I20251026 02:56:53.095559 2372 log.cc:1199] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/3bcd7dec9c2149069e48dcf5e564595e
I20251026 02:56:53.095916 2372 ts_tablet_manager.cc:1950] T 3bcd7dec9c2149069e48dcf5e564595e P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:53.097311 2226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d30f806bf2a4c3685d8398be78e91d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.100334 2234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073b67cccd6b47fdbd64917267c0b4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.101848 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1086) took 6227 ms. Trace:
I20251026 02:56:53.101931 2372 rpcz_store.cc:276] 1026 02:56:46.874807 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:46.874874 (+ 67us) service_pool.cc:225] Handling call
1026 02:56:52.634771 (+5759897us) ts_tablet_manager.cc:1151] Deleting tablet 3bcd7dec9c2149069e48dcf5e564595e
1026 02:56:52.634783 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.693773 (+ 58990us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:52.951032 (+257259us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:53.101825 (+150793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5759696}
I20251026 02:56:53.102363 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1090) took 5964 ms. Trace:
I20251026 02:56:53.102432 2372 rpcz_store.cc:276] 1026 02:56:47.137950 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.137991 (+ 41us) service_pool.cc:225] Handling call
1026 02:56:53.102234 (+5964243us) ts_tablet_manager.cc:1151] Deleting tablet f80b5c4c4e2d4c14a8a5292a05127bc1
1026 02:56:53.102245 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:53.102345 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5963506}
I20251026 02:56:53.102731 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1093) took 5806 ms. Trace:
I20251026 02:56:53.102783 2372 rpcz_store.cc:276] 1026 02:56:47.296419 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.299371 (+ 2952us) service_pool.cc:225] Handling call
1026 02:56:53.102635 (+5803264us) ts_tablet_manager.cc:1151] Deleting tablet 68ad3d4a2d484badb4b35884517b72f1
1026 02:56:53.102643 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:53.102721 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5803037}
I20251026 02:56:53.104140 2228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aedb0d839b44ac887e769c593900f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.106514 4484 raft_consensus.cc:493] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:53.106671 4484 raft_consensus.cc:515] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.107378 4484 leader_election.cc:290] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:53.107450 2239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8423467856494a78acc7d5498b9f5506. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.111380 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c0d1b731074ee1acc9c3032c8bef71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.111882 4482 consensus_queue.cc:1048] T b8fe7825fa8d46669772cf84bc4b3afc P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:53.112246 4482 consensus_queue.cc:1048] T a1d8bfca6a554e2abbbb9a0c7e2c0c94 P 696476265495492790739ec126a08539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 02:56:53.115626 4069 raft_consensus.cc:697] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:53.115741 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet 3bcd7dec9c2149069e48dcf5e564595e (table default.Table_7eb7fd8131cc4bce81b6cc09c5062fc0 [id=5cbfb8198a8947b3abd1c0801e92f100]) successfully deleted
W20251026 02:56:53.115981 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet f80b5c4c4e2d4c14a8a5292a05127bc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f80b5c4c4e2d4c14a8a5292a05127bc1 already in progress: creating tablet
I20251026 02:56:53.115916 4069 consensus_queue.cc:237] T 3ac38e6257e74367b06b15fc49d2af81 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:53.116161 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 68ad3d4a2d484badb4b35884517b72f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68ad3d4a2d484badb4b35884517b72f1 already in progress: creating tablet
I20251026 02:56:53.116833 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67cb4a4a87a44304a132cc0fc0bf7bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.117306 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ad380f321d514c00ad865a0efe4893e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:53.117851 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad380f321d514c00ad865a0efe4893e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.119045 1660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_3abedac04a774450868110f2a7d9ac3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:53.119479 1660 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3abedac04a774450868110f2a7d9ac3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.120160 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d81edc20febc45618393e9dc810d7c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.120678 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_f2e34abc2f4148f390dd8ccfda7e4fb8" } new_table_name: "default.Table_87cd63efdcc3486698e6d6f7e9a36fea" modify_external_catalogs: true
I20251026 02:56:53.122107 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_35a3e6169a8346a497f84b70512a1fcb" } modify_external_catalogs: true
I20251026 02:56:53.122233 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_35a3e6169a8346a497f84b70512a1fcb" } modify_external_catalogs: true
W20251026 02:56:53.125126 2053 leader_election.cc:343] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.125250 2053 leader_election.cc:304] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.125569 4098 raft_consensus.cc:493] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:53.125628 2053 leader_election.cc:304] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.125680 4098 raft_consensus.cc:515] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.126147 4428 raft_consensus.cc:2749] T 28dfe285139042349b5b9d6cbc59a778 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:53.126192 4098 leader_election.cc:290] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.126303 4098 raft_consensus.cc:2804] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:53.126363 4098 raft_consensus.cc:493] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:53.126417 4098 raft_consensus.cc:3060] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.128566 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 7eb85291b9fd4fed92139084df5eed60
I20251026 02:56:53.129172 1658 master_service.cc:954] ReplaceTablet: received request to replace tablet 1ad6906bb58a4d5fbbb64c5bc6acef8a from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:53.131166 4098 raft_consensus.cc:515] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.131781 4098 leader_election.cc:290] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:53.132884 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d01fa8b270e1476cb86d81a196ba94f8 consensus state has no leader
I20251026 02:56:53.133505 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 7eb85291b9fd4fed92139084df5eed60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:52 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:53.134166 4427 leader_election.cc:290] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.134928 4488 tablet.cc:1722] T cfac07917ee34e6dba3fac5e081f6b43 P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251026 02:56:53.020121 1915 leader_election.cc:304] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
W20251026 02:56:53.140543 1915 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 120430us
I20251026 02:56:53.141311 4458 raft_consensus.cc:2804] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:53.141417 4458 raft_consensus.cc:493] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:53.141503 4458 raft_consensus.cc:3060] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.028872 1981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892a5fab2af1432eb31d92d9859ea2a0" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:53.142832 2194 leader_election.cc:343] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.006788 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1091) took 4015 ms. Trace:
I20251026 02:56:53.142976 2194 leader_election.cc:304] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 696476265495492790739ec126a08539; no voters: 0b6b0a6c4e144982b67980118cddba48, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.143028 2376 rpcz_store.cc:276] 1026 02:56:48.991113 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.991186 (+ 73us) service_pool.cc:225] Handling call
1026 02:56:52.615649 (+3624463us) ts_tablet_manager.cc:1151] Deleting tablet 1fb2ff87cf0248a0a4a77f2733ef867b
1026 02:56:52.615657 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.924194 (+308537us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:53.005850 (+ 81656us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:53.006755 (+ 905us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":193432,"tablet-delete.queue_time_us":3624241}
I20251026 02:56:53.143474 2376 tablet_replica.cc:333] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:53.143589 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 1fb2ff87cf0248a0a4a77f2733ef867b (table default.Table_26257eca63ce4baaa75beae2e9db1340 [id=fbc97817a9584474abc0b2ab2052abb2]) successfully deleted
I20251026 02:56:53.143738 2376 raft_consensus.cc:2243] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:53.036808 1957 tablet_service.cc:1552] Processing DeleteTablet for tablet 89a7853ff92c4350822b205a322fc62b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:52 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:53.144063 4487 raft_consensus.cc:2749] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 02:56:53.144109 2376 raft_consensus.cc:2272] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:53.144702 2376 ts_tablet_manager.cc:1916] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:53.144834 4458 raft_consensus.cc:515] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.145534 4458 leader_election.cc:290] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.046164 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 09f5265d85c6496d85c555b2b847373a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:53.145721 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_132762ab807d477b8d557333fb730eae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:53.046613 1958 tablet_service.cc:1552] Processing DeleteTablet for tablet 283939e86520421b839f9c1af0ddeba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:51 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:53.083292 1966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24f16e40a30c41788a6f9813c235c5ab" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:53.146291 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_132762ab807d477b8d557333fb730eae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.126705 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892a5fab2af1432eb31d92d9859ea2a0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:53.134878 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7834a85168364390a841fc883681c845" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:53.141306 1915 consensus_peers.cc:597] T f2d97084b5914350afbfda5c2a34bf9a P 7fef01f4f2b843538ea245a09dd087a8 -> Peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Couldn't send request to peer 696476265495492790739ec126a08539. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2d97084b5914350afbfda5c2a34bf9a. This is attempt 1: this message will repeat every 5th retry.
W20251026 02:56:53.147011 1915 leader_election.cc:343] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 4a3f3fc3c7a04d05bae28472251f3acc
I20251026 02:56:53.147114 1915 leader_election.cc:304] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
W20251026 02:56:53.147466 1915 leader_election.cc:343] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: dfd3e23d83a84648bd11cd258377aecb
I20251026 02:56:53.147578 1915 leader_election.cc:304] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:53.147634 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9f6ae0bb97e0495895985e078c5e79b0" } new_table_name: "default.Table_e9dd1bf70df24007b55d0a45555574e6" modify_external_catalogs: true
I20251026 02:56:53.148365 4460 raft_consensus.cc:2749] T 4a3f3fc3c7a04d05bae28472251f3acc P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:53.148527 4460 raft_consensus.cc:2749] T dfd3e23d83a84648bd11cd258377aecb P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:53.150228 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87895fb88c284a4f8d7c58a5ee48ba8e" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:53.150427 2256 raft_consensus.cc:3060] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:53.150763 1916 leader_election.cc:343] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 87895fb88c284a4f8d7c58a5ee48ba8e
I20251026 02:56:53.150966 4428 raft_consensus.cc:493] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:53.151111 4428 raft_consensus.cc:515] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:53.151543 2194 leader_election.cc:343] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.151654 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7834a85168364390a841fc883681c845" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:53.151856 2118 raft_consensus.cc:2468] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:53.151723 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1094) took 5758 ms. Trace:
I20251026 02:56:53.152319 2372 rpcz_store.cc:276] 1026 02:56:47.393035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.396371 (+ 3336us) service_pool.cc:225] Handling call
1026 02:56:53.102942 (+5706571us) ts_tablet_manager.cc:1151] Deleting tablet 0aabfe11dedd454f8221fb5de25a268b
1026 02:56:53.102948 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:53.151680 (+ 48732us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5706415}
W20251026 02:56:53.152755 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 0aabfe11dedd454f8221fb5de25a268b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0aabfe11dedd454f8221fb5de25a268b already in progress: creating tablet
I20251026 02:56:53.153076 2372 tablet_replica.cc:333] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:53.153046 2256 raft_consensus.cc:2468] T 87895fb88c284a4f8d7c58a5ee48ba8e P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fef01f4f2b843538ea245a09dd087a8 in term 1.
I20251026 02:56:53.153319 2372 raft_consensus.cc:2243] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:53.154175 1915 leader_election.cc:304] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48
I20251026 02:56:53.154613 4460 raft_consensus.cc:2804] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:53.154753 4460 raft_consensus.cc:697] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Running, Role: LEADER
I20251026 02:56:53.151659 4428 leader_election.cc:290] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.154989 4460 consensus_queue.cc:237] T 87895fb88c284a4f8d7c58a5ee48ba8e P 7fef01f4f2b843538ea245a09dd087a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.042042 2378 tablet_bootstrap.cc:492] T 6699b50d87964f6a89c365171253ad25 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:53.158968 2378 ts_tablet_manager.cc:1403] T 6699b50d87964f6a89c365171253ad25 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.353s user 0.012s sys 0.027s
I20251026 02:56:53.159894 2378 raft_consensus.cc:359] T 6699b50d87964f6a89c365171253ad25 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.160997 2378 raft_consensus.cc:385] T 6699b50d87964f6a89c365171253ad25 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:53.160967 2195 leader_election.cc:304] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.161347 2378 raft_consensus.cc:740] T 6699b50d87964f6a89c365171253ad25 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:53.161666 4479 raft_consensus.cc:2804] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:53.161777 4479 raft_consensus.cc:493] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:53.161849 4479 raft_consensus.cc:3060] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.163650 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 7eb85291b9fd4fed92139084df5eed60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:52 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.163923 4098 raft_consensus.cc:493] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:53.164111 2375 tablet_bootstrap.cc:492] T 1a1b8c440061429caea0f0803f110742 P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:53.164069 4098 raft_consensus.cc:515] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.164295 2375 ts_tablet_manager.cc:1403] T 1a1b8c440061429caea0f0803f110742 P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.004s
I20251026 02:56:53.164669 4098 leader_election.cc:290] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.164763 4479 raft_consensus.cc:515] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.165247 2375 raft_consensus.cc:359] T 1a1b8c440061429caea0f0803f110742 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.165441 2375 raft_consensus.cc:385] T 1a1b8c440061429caea0f0803f110742 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:53.165532 2375 raft_consensus.cc:740] T 1a1b8c440061429caea0f0803f110742 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:53.165761 2375 consensus_queue.cc:260] T 1a1b8c440061429caea0f0803f110742 P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.166965 4479 leader_election.cc:290] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:53.165777 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7834a85168364390a841fc883681c845" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:53.167414 2194 leader_election.cc:343] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.167450 2378 consensus_queue.cc:260] T 6699b50d87964f6a89c365171253ad25 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.167560 2118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7834a85168364390a841fc883681c845" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:53.167774 2118 raft_consensus.cc:3060] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.167970 2378 ts_tablet_manager.cc:1434] T 6699b50d87964f6a89c365171253ad25 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251026 02:56:53.168607 2378 tablet_bootstrap.cc:492] T 868dbbfce40c4dd6b36cc220966b1986 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:53.166028 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 7eb85291b9fd4fed92139084df5eed60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:52 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:53.169552 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 43938.9us
I20251026 02:56:53.169754 2053 leader_election.cc:304] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.170208 4428 raft_consensus.cc:2804] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:53.170302 4428 raft_consensus.cc:493] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:53.170388 4428 raft_consensus.cc:3060] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.170819 2118 raft_consensus.cc:2468] T 7834a85168364390a841fc883681c845 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:53.171883 2195 leader_election.cc:304] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.172156 4479 raft_consensus.cc:2804] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:53.172335 4479 raft_consensus.cc:697] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [term 1 LEADER]: Becoming Leader. State: Replica: 696476265495492790739ec126a08539, State: Running, Role: LEADER
I20251026 02:56:53.172610 4479 consensus_queue.cc:237] T 7834a85168364390a841fc883681c845 P 696476265495492790739ec126a08539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.172914 4428 raft_consensus.cc:515] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.173936 4428 leader_election.cc:290] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.176155 2053 leader_election.cc:304] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters:
I20251026 02:56:53.176465 2053 leader_election.cc:304] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.176729 4427 raft_consensus.cc:2804] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:53.176849 4427 raft_consensus.cc:493] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:53.176931 4427 raft_consensus.cc:3060] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.176996 2375 ts_tablet_manager.cc:1434] T 1a1b8c440061429caea0f0803f110742 P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251026 02:56:53.177371 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aa7a90a29ec430286786bf8aa9d4c02" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:53.177856 4098 raft_consensus.cc:2804] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:53.177938 4098 raft_consensus.cc:493] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:53.178022 4098 raft_consensus.cc:3060] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.178360 2377 ts_tablet_manager.cc:1929] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:53.178447 2377 log.cc:1199] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/a53f25688e3b45dca68984e2c543d094
I20251026 02:56:53.178750 2377 ts_tablet_manager.cc:1950] T a53f25688e3b45dca68984e2c543d094 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:53.179406 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1176) took 3916 ms. Trace:
I20251026 02:56:53.179315 4427 raft_consensus.cc:515] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.179893 4427 leader_election.cc:290] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.181936 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1b6613742a24c21a1db6ebe858a846f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:53.182802 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c966cae6886048a2b6ad3b8bd7007bcb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:53.183521 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db94738da3554fbda64c23fde3ef6c1b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:53.184456 2375 tablet_bootstrap.cc:492] T 76c3e5fc281847f484ae71a19336a91c P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:53.184520 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81775055d98349f696cdab6171ffd58b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:53.185519 4098 raft_consensus.cc:515] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.186489 4098 leader_election.cc:290] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:53.189085 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3aea1a654314c5fb6973c33a142627c consensus state has no leader
I20251026 02:56:53.189443 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_b2484a84a6fc475bad4555f677c7b009"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:53.189970 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2484a84a6fc475bad4555f677c7b009 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.191620 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_2cc3dec7723041bb8cd4b51cd2a05cb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:53.192106 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cc3dec7723041bb8cd4b51cd2a05cb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251026 02:56:53.193435 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04265a5178034dcaa2bb14b1efa3e493 consensus state has no leader
I20251026 02:56:53.193765 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_12c63af0fb4a44b19e48f3ecb496a2d2" } new_table_name: "default.Table_2d21d400c25e4e0c830014e1cc7f8295" modify_external_catalogs: true
I20251026 02:56:53.195204 1660 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_3fb4e68115eb4b9f98a57da13662a443" } new_table_name: "default.Table_18a390fc36384658ac0f21b9282ec9b0" modify_external_catalogs: true
I20251026 02:56:53.196825 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a7a5020acff04c1d8b5749cdfd7dddfd
I20251026 02:56:53.197341 1654 master_service.cc:954] ReplaceTablet: received request to replace tablet 8862cd48a5744c99b32f365cb77fa7c4 from {username='slave'} at 127.0.0.1:39910
I20251026 02:56:53.202620 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet a7a5020acff04c1d8b5749cdfd7dddfd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:53.203267 2372 raft_consensus.cc:2272] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:53.203678 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet a7a5020acff04c1d8b5749cdfd7dddfd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:53.204113 2372 ts_tablet_manager.cc:1916] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:53.206262 2378 tablet_bootstrap.cc:654] T 868dbbfce40c4dd6b36cc220966b1986 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:53.210542 2375 tablet_bootstrap.cc:654] T 76c3e5fc281847f484ae71a19336a91c P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:53.214659 2378 tablet_bootstrap.cc:492] T 868dbbfce40c4dd6b36cc220966b1986 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:53.214974 2378 ts_tablet_manager.cc:1403] T 868dbbfce40c4dd6b36cc220966b1986 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20251026 02:56:53.215723 2378 raft_consensus.cc:359] T 868dbbfce40c4dd6b36cc220966b1986 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.216564 2378 raft_consensus.cc:385] T 868dbbfce40c4dd6b36cc220966b1986 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:53.216696 2378 raft_consensus.cc:740] T 868dbbfce40c4dd6b36cc220966b1986 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:53.216981 2378 consensus_queue.cc:260] T 868dbbfce40c4dd6b36cc220966b1986 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.217622 2378 ts_tablet_manager.cc:1434] T 868dbbfce40c4dd6b36cc220966b1986 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251026 02:56:53.221589 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_06ce70931ac34135accc9d6cc0bf5862"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:53.222144 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06ce70931ac34135accc9d6cc0bf5862 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.224472 2378 tablet_bootstrap.cc:492] T 892a5fab2af1432eb31d92d9859ea2a0 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:53.225800 4443 tablet.cc:1722] T 882cd1680aa140f6a63a961cd4aec25a P 7fef01f4f2b843538ea245a09dd087a8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:53.227368 2378 tablet_bootstrap.cc:654] T 892a5fab2af1432eb31d92d9859ea2a0 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:53.235069 2377 rpcz_store.cc:276] 1026 02:56:49.262645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:49.263130 (+ 485us) service_pool.cc:225] Handling call
1026 02:56:52.721766 (+3458636us) ts_tablet_manager.cc:1151] Deleting tablet a53f25688e3b45dca68984e2c543d094
1026 02:56:52.721776 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:52.970262 (+248486us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:53.178251 (+207989us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:53.179387 (+ 1136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2745,"tablet-delete.queue_time_us":3412241}
I20251026 02:56:53.235867 2377 tablet_replica.cc:333] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:53.236202 2377 raft_consensus.cc:2243] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:53.236687 2377 raft_consensus.cc:2272] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:53.237414 2377 ts_tablet_manager.cc:1916] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:53.211323 2053 leader_election.cc:343] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:53.242225 2053 leader_election.cc:343] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:53.242514 2053 leader_election.cc:343] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:53.242812 2053 leader_election.cc:343] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:53.243003 2053 leader_election.cc:343] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:53.243201 2053 leader_election.cc:343] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:53.243435 2053 leader_election.cc:343] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.244029 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24f16e40a30c41788a6f9813c235c5ab" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:53.226320 4488 tablet.cc:1722] T 882cd1680aa140f6a63a961cd4aec25a P 696476265495492790739ec126a08539: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251026 02:56:53.213171 2381 ts_tablet_manager.cc:1434] T 09f5265d85c6496d85c555b2b847373a P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.140s user 0.002s sys 0.004s
I20251026 02:56:53.245354 2381 tablet_bootstrap.cc:492] T 027dd92fc5cf4c9e847de3e55e0dac1c P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:53.246547 4458 raft_consensus.cc:493] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 696476265495492790739ec126a08539)
I20251026 02:56:53.246703 4458 raft_consensus.cc:515] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.247025 2381 tablet_bootstrap.cc:654] T 027dd92fc5cf4c9e847de3e55e0dac1c P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:53.248255 4458 leader_election.cc:290] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.248651 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09f5265d85c6496d85c555b2b847373a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:53.249387 1916 leader_election.cc:343] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: d43b6f22dd4f4f9c9657f21514550312
I20251026 02:56:53.254451 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet a7a5020acff04c1d8b5749cdfd7dddfd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.255185 2259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1b6613742a24c21a1db6ebe858a846f" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:53.255313 4485 raft_consensus.cc:493] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:53.255373 2259 raft_consensus.cc:3060] T b1b6613742a24c21a1db6ebe858a846f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.255429 4485 raft_consensus.cc:515] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.255941 4485 leader_election.cc:290] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:53.256736 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ab6a6bea093e4725896563846bea74a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:53.257258 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab6a6bea093e4725896563846bea74a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.257788 2259 raft_consensus.cc:2468] T b1b6613742a24c21a1db6ebe858a846f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:53.258823 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_ae81c912abfe4af688b02747a0a2bbe0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:53.259400 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae81c912abfe4af688b02747a0a2bbe0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.259425 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "892a5fab2af1432eb31d92d9859ea2a0" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:53.259536 4490 raft_consensus.cc:493] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:53.259614 2256 raft_consensus.cc:2393] T 892a5fab2af1432eb31d92d9859ea2a0 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b6b0a6c4e144982b67980118cddba48 in current term 1: Already voted for candidate 696476265495492790739ec126a08539 in this term.
I20251026 02:56:53.259661 4490 raft_consensus.cc:515] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.260174 4490 leader_election.cc:290] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:53.260195 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c966cae6886048a2b6ad3b8bd7007bcb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:53.260388 2256 raft_consensus.cc:2468] T c966cae6886048a2b6ad3b8bd7007bcb P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
W20251026 02:56:53.260560 1663 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 706036515b454880b3413a994c5d1138 consensus state has no leader
I20251026 02:56:53.260802 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db94738da3554fbda64c23fde3ef6c1b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:53.260913 2256 raft_consensus.cc:2468] T db94738da3554fbda64c23fde3ef6c1b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 0.
I20251026 02:56:53.260912 1663 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2d20995352a745b1a63b0cb745126074" } modify_external_catalogs: true
I20251026 02:56:53.261071 1663 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_2d20995352a745b1a63b0cb745126074" } modify_external_catalogs: true
I20251026 02:56:53.261327 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81775055d98349f696cdab6171ffd58b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:53.261435 2256 raft_consensus.cc:3060] T 81775055d98349f696cdab6171ffd58b P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
W20251026 02:56:53.261703 1660 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1807c0737d744d4cab29393e9122ff6b consensus state has no leader
I20251026 02:56:53.262056 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_217ad44ad25140129bf8e39c88b3b593" } modify_external_catalogs: true
I20251026 02:56:53.262205 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_217ad44ad25140129bf8e39c88b3b593" } modify_external_catalogs: true
W20251026 02:56:53.263590 2053 leader_election.cc:343] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:53.263793 2053 leader_election.cc:343] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.264263 1966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f0124f125c40d291c2211fd95c71c5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:53.265018 2256 raft_consensus.cc:2468] T 81775055d98349f696cdab6171ffd58b P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
W20251026 02:56:53.265031 2194 leader_election.cc:343] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.265668 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24f16e40a30c41788a6f9813c235c5ab" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:53.265849 2256 raft_consensus.cc:3060] T 24f16e40a30c41788a6f9813c235c5ab P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.265808 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_d80792284e314e01924afa248daa6afa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:53.266148 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 8862cd48a5744c99b32f365cb77fa7c4
W20251026 02:56:53.266425 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d80792284e314e01924afa248daa6afa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.266597 1654 catalog_manager.cc:6719] ReplaceTablet: tablet 8862cd48a5744c99b32f365cb77fa7c4 deleted and replaced by tablet b4794b770c504468babc3b2cf289a4c6
I20251026 02:56:53.266868 1654 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7a245074eac243e29b301b5f1c84a32d" } modify_external_catalogs: true
I20251026 02:56:53.267001 1654 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_7a245074eac243e29b301b5f1c84a32d" } modify_external_catalogs: true
I20251026 02:56:53.267733 2094 tablet_service.cc:1552] Processing DeleteTablet for tablet 8862cd48a5744c99b32f365cb77fa7c4 with delete_type TABLET_DATA_DELETED (replaced by tablet b4794b770c504468babc3b2cf289a4c6) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:53.268834 2256 raft_consensus.cc:2468] T 24f16e40a30c41788a6f9813c235c5ab P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:53.269071 1961 tablet_service.cc:1552] Processing DeleteTablet for tablet 8862cd48a5744c99b32f365cb77fa7c4 with delete_type TABLET_DATA_DELETED (replaced by tablet b4794b770c504468babc3b2cf289a4c6) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:53.269731 2256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09f5265d85c6496d85c555b2b847373a" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539"
I20251026 02:56:53.269928 2256 raft_consensus.cc:3060] T 09f5265d85c6496d85c555b2b847373a P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.270570 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79c18ebf56e045b6a52fae8871a9107f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
W20251026 02:56:53.271126 2194 leader_election.cc:343] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.272698 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aa7a90a29ec430286786bf8aa9d4c02" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:53.272933 2256 raft_consensus.cc:2468] T 09f5265d85c6496d85c555b2b847373a P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6b0a6c4e144982b67980118cddba48 in term 1.
I20251026 02:56:53.275223 1647 catalog_manager.cc:4985] TS 696476265495492790739ec126a08539 (127.1.150.3:38107): tablet a53f25688e3b45dca68984e2c543d094 (table default.Table_feb822201306444ebe478734d286a667 [id=535d3143b66249a083853a11934d994e]) successfully deleted
I20251026 02:56:53.274381 2378 tablet_bootstrap.cc:492] T 892a5fab2af1432eb31d92d9859ea2a0 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:53.275553 2378 ts_tablet_manager.cc:1403] T 892a5fab2af1432eb31d92d9859ea2a0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20251026 02:56:53.276321 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79c18ebf56e045b6a52fae8871a9107f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:53.276307 2378 raft_consensus.cc:359] T 892a5fab2af1432eb31d92d9859ea2a0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.276490 2378 raft_consensus.cc:385] T 892a5fab2af1432eb31d92d9859ea2a0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:53.276588 2378 raft_consensus.cc:740] T 892a5fab2af1432eb31d92d9859ea2a0 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:53.276844 2116 raft_consensus.cc:2468] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:53.276858 2378 consensus_queue.cc:260] T 892a5fab2af1432eb31d92d9859ea2a0 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.277355 2378 ts_tablet_manager.cc:1434] T 892a5fab2af1432eb31d92d9859ea2a0 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:53.277668 2378 tablet_bootstrap.cc:492] T 3ac38e6257e74367b06b15fc49d2af81 P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:53.277944 2375 tablet_bootstrap.cc:492] T 76c3e5fc281847f484ae71a19336a91c P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:53.278085 2375 ts_tablet_manager.cc:1403] T 76c3e5fc281847f484ae71a19336a91c P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.094s user 0.005s sys 0.000s
I20251026 02:56:53.279145 2375 raft_consensus.cc:359] T 76c3e5fc281847f484ae71a19336a91c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.279337 2375 raft_consensus.cc:385] T 76c3e5fc281847f484ae71a19336a91c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:53.279412 2375 raft_consensus.cc:740] T 76c3e5fc281847f484ae71a19336a91c P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:53.279657 2375 consensus_queue.cc:260] T 76c3e5fc281847f484ae71a19336a91c P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.282464 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f0124f125c40d291c2211fd95c71c5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:53.282733 2116 raft_consensus.cc:2468] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:53.283835 2053 leader_election.cc:304] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.284363 4494 raft_consensus.cc:2804] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:53.284565 4494 raft_consensus.cc:697] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:53.284809 4494 consensus_queue.cc:237] T b1b6613742a24c21a1db6ebe858a846f P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.289947 2053 leader_election.cc:304] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.290266 4098 raft_consensus.cc:2749] T 892a5fab2af1432eb31d92d9859ea2a0 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 02:56:53.290802 2053 leader_election.cc:304] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.291147 4098 raft_consensus.cc:2804] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:53.291237 4098 raft_consensus.cc:493] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:53.291321 4098 raft_consensus.cc:3060] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.294092 4479 raft_consensus.cc:493] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:53.294456 4479 raft_consensus.cc:515] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.294776 4098 raft_consensus.cc:515] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.295095 4485 raft_consensus.cc:493] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:53.295243 4485 raft_consensus.cc:515] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.295672 4098 leader_election.cc:290] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.295675 4479 leader_election.cc:290] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:53.295992 4485 leader_election.cc:290] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:53.297217 2381 tablet_bootstrap.cc:492] T 027dd92fc5cf4c9e847de3e55e0dac1c P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:53.297369 2381 ts_tablet_manager.cc:1403] T 027dd92fc5cf4c9e847de3e55e0dac1c P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.003s
I20251026 02:56:53.299005 2053 leader_election.cc:304] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.299325 4098 raft_consensus.cc:2804] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:53.299408 4098 raft_consensus.cc:493] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:53.299481 4098 raft_consensus.cc:3060] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.303718 4098 raft_consensus.cc:515] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.304746 4098 leader_election.cc:290] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.279378 2378 tablet_bootstrap.cc:654] T 3ac38e6257e74367b06b15fc49d2af81 P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:53.317806 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 398c1f420974453b9b6b0fe17795378b (table default.Table_a92e68d05f4b455d810180ee4cc5874d [id=7f64fbec3582415a8924d01e14856a56]). Expected version 1 got 0
I20251026 02:56:53.318287 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet db94738da3554fbda64c23fde3ef6c1b (table default.Table_f9afa2d0968b401e8fc615844c100b5e [id=928effe263004633a026f61a978a1e8a]). Expected version 2 got 0
I20251026 02:56:53.318532 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 31b6e08597bd450eb1875201cf594a82 (table default.Table_436964d7d7d54f38a72eebf1d65d8d6e [id=a557708372bd4249969f80adf8641556]). Expected version 3 got 0
I20251026 02:56:53.318714 1661 catalog_manager.cc:5649] T 3aea01a248b345cf81d4d9862744eb78 P 696476265495492790739ec126a08539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696476265495492790739ec126a08539 (127.1.150.3). New cstate: current_term: 1 leader_uuid: "696476265495492790739ec126a08539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } health_report { overall_health: HEALTHY } } }
I20251026 02:56:53.319057 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 3aea01a248b345cf81d4d9862744eb78 (table default.Table_d68a91a3695240a39e5526b9cff6f6f2 [id=74d6732406a448e49911361bdc64417a]). Expected version 2 got 0
I20251026 02:56:53.319336 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 41a1442fffe54add93666e3c78a0cc58 (table default.Table_1faa0c8151724c5e816546ee2e1d5182 [id=a106f9ece0b9467a8bc1d5d4b3524c41]). Expected version 1 got 0
I20251026 02:56:53.319453 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet c966cae6886048a2b6ad3b8bd7007bcb (table default.Table_8dadea9fa7254668b84991c8d7cc4f52 [id=f7982d61f7d8411193899caaa357b61b]). Expected version 1 got 0
I20251026 02:56:53.322947 2053 leader_election.cc:304] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.323449 2053 leader_election.cc:304] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.323746 4427 raft_consensus.cc:2804] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:53.323894 4427 raft_consensus.cc:697] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:53.323940 4428 raft_consensus.cc:2804] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:53.324186 4427 consensus_queue.cc:237] T 81775055d98349f696cdab6171ffd58b P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.328480 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_6ac689880a3540e0a0e5a139702a7dd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:53.329006 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ac689880a3540e0a0e5a139702a7dd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.331450 1961 tablet_service.cc:1505] Processing CreateTablet for tablet 5ba7031c2e574617b88a597357b873a0 (DEFAULT_TABLE table=default.Table_0e87fae7410549a88fc45e95876f13ea [id=f2d54ae909294290b3a6ad748494411a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.332399 1657 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4e1e3945b1094e179d628f897d2241dc" } modify_external_catalogs: true
I20251026 02:56:53.332564 1657 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_4e1e3945b1094e179d628f897d2241dc" } modify_external_catalogs: true
I20251026 02:56:53.335093 2094 tablet_service.cc:1505] Processing CreateTablet for tablet 5ba7031c2e574617b88a597357b873a0 (DEFAULT_TABLE table=default.Table_0e87fae7410549a88fc45e95876f13ea [id=f2d54ae909294290b3a6ad748494411a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.335414 1659 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_dbd6c27f936b44f48242fff9d1b6ec82" } new_table_name: "default.Table_3749112c4de747afb527f7a04fdf271e" modify_external_catalogs: true
I20251026 02:56:53.335613 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba7031c2e574617b88a597357b873a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.336755 4376 raft_consensus.cc:493] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 696476265495492790739ec126a08539)
I20251026 02:56:53.336907 4376 raft_consensus.cc:515] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.337990 4376 leader_election.cc:290] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.339625 1958 tablet_service.cc:1505] Processing CreateTablet for tablet cfff2174bac74ab0999215c2aed5b955 (DEFAULT_TABLE table=default.Table_39cd69856c0a4284b604da1e447e172a [id=1e8834dad64e4a2f9266f10948603710]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.339269 2381 raft_consensus.cc:359] T 027dd92fc5cf4c9e847de3e55e0dac1c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.340008 2381 raft_consensus.cc:385] T 027dd92fc5cf4c9e847de3e55e0dac1c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:53.340058 2381 raft_consensus.cc:740] T 027dd92fc5cf4c9e847de3e55e0dac1c P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
W20251026 02:56:53.341105 1916 leader_election.cc:343] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 95c9fbae62db4850b0f02a2443a3432b
I20251026 02:56:53.344427 2381 consensus_queue.cc:260] T 027dd92fc5cf4c9e847de3e55e0dac1c P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.345165 2094 tablet_service.cc:1505] Processing CreateTablet for tablet cfff2174bac74ab0999215c2aed5b955 (DEFAULT_TABLE table=default.Table_39cd69856c0a4284b604da1e447e172a [id=1e8834dad64e4a2f9266f10948603710]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.345813 2094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfff2174bac74ab0999215c2aed5b955. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.346554 1658 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 1ad6906bb58a4d5fbbb64c5bc6acef8a
I20251026 02:56:53.346916 1658 catalog_manager.cc:6719] ReplaceTablet: tablet 1ad6906bb58a4d5fbbb64c5bc6acef8a deleted and replaced by tablet 5c9addce32ec451484003598c4f4592e
I20251026 02:56:53.347677 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ad6906bb58a4d5fbbb64c5bc6acef8a with delete_type TABLET_DATA_DELETED (replaced by tablet 5c9addce32ec451484003598c4f4592e) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:53.348165 1660 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 6d06944f71fd4e42a6ea46969c86e9f1
I20251026 02:56:53.349372 1944 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d06944f71fd4e42a6ea46969c86e9f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:53.347514 2098 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ad6906bb58a4d5fbbb64c5bc6acef8a with delete_type TABLET_DATA_DELETED (replaced by tablet 5c9addce32ec451484003598c4f4592e) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:53.349555 2093 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d06944f71fd4e42a6ea46969c86e9f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:53.353700 1944 tablet_service.cc:1505] Processing CreateTablet for tablet 37a34c28786c4cebb741f8218a72502b (DEFAULT_TABLE table=default.Table_fdaf051a8a4b4899a92c293e6428e200 [id=129a0148a075452aa137277489f0ac10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.356063 4376 raft_consensus.cc:493] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:53.356781 4376 raft_consensus.cc:515] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.357609 2375 ts_tablet_manager.cc:1434] T 76c3e5fc281847f484ae71a19336a91c P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.079s user 0.001s sys 0.001s
I20251026 02:56:53.357837 4376 leader_election.cc:290] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.358239 4428 raft_consensus.cc:697] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:53.358609 4428 consensus_queue.cc:237] T 24f16e40a30c41788a6f9813c235c5ab P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.360147 2093 tablet_service.cc:1505] Processing CreateTablet for tablet 37a34c28786c4cebb741f8218a72502b (DEFAULT_TABLE table=default.Table_fdaf051a8a4b4899a92c293e6428e200 [id=129a0148a075452aa137277489f0ac10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.360584 2093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37a34c28786c4cebb741f8218a72502b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.345001 2381 ts_tablet_manager.cc:1434] T 027dd92fc5cf4c9e847de3e55e0dac1c P 696476265495492790739ec126a08539: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20251026 02:56:53.362090 2381 tablet_bootstrap.cc:492] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 696476265495492790739ec126a08539: Bootstrap starting.
I20251026 02:56:53.364051 2381 tablet_bootstrap.cc:654] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 696476265495492790739ec126a08539: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:53.367710 2098 tablet_service.cc:1505] Processing CreateTablet for tablet 01b2eeedec7f48088b54d79394432dd7 (DEFAULT_TABLE table=default.Table_34885bd9ba024c96a4bea0c2fedcfb4b [id=94a166ea5a59425797bb4af071a97abf]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:53.364233 2195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.150.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 66578.8us
I20251026 02:56:53.368664 2195 leader_election.cc:304] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.369032 4255 raft_consensus.cc:2804] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:53.369102 4255 raft_consensus.cc:493] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:53.369158 4255 raft_consensus.cc:3060] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.370440 1957 tablet_service.cc:1505] Processing CreateTablet for tablet 01b2eeedec7f48088b54d79394432dd7 (DEFAULT_TABLE table=default.Table_34885bd9ba024c96a4bea0c2fedcfb4b [id=94a166ea5a59425797bb4af071a97abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.371913 1654 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 72e4a4ea953f4f2786a39c0ddbd11219
I20251026 02:56:53.371925 4255 raft_consensus.cc:515] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.373039 4255 leader_election.cc:290] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:53.373023 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79c18ebf56e045b6a52fae8871a9107f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:53.369185 2195 leader_election.cc:304] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.373479 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet 72e4a4ea953f4f2786a39c0ddbd11219 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:53.373651 4255 raft_consensus.cc:2804] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 02:56:53.373760 4255 raft_consensus.cc:493] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 02:56:53.373862 4255 raft_consensus.cc:3060] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.373888 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet 72e4a4ea953f4f2786a39c0ddbd11219 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:53.373934 2194 leader_election.cc:343] T 79c18ebf56e045b6a52fae8871a9107f P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.375097 1663 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet a02c9d84f0ec498e896ec97a75e3c154
I20251026 02:56:53.375926 2088 tablet_service.cc:1552] Processing DeleteTablet for tablet a02c9d84f0ec498e896ec97a75e3c154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:53.376492 1962 tablet_service.cc:1552] Processing DeleteTablet for tablet a02c9d84f0ec498e896ec97a75e3c154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:53.376610 1916 leader_election.cc:343] T 7f22c6c163c34ed48a0b7eeb5a574e76 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): Not found: Tablet not found: 7f22c6c163c34ed48a0b7eeb5a574e76
I20251026 02:56:53.377170 4255 raft_consensus.cc:515] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.378163 4255 leader_election.cc:290] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Requested vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611)
I20251026 02:56:53.378141 1982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f0124f125c40d291c2211fd95c71c5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:53.378621 2194 leader_election.cc:343] T f8f0124f125c40d291c2211fd95c71c5 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.379215 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 8862cd48a5744c99b32f365cb77fa7c4 with delete_type TABLET_DATA_DELETED (replaced by tablet b4794b770c504468babc3b2cf289a4c6) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.379629 2224 tablet_service.cc:1505] Processing CreateTablet for tablet 5ba7031c2e574617b88a597357b873a0 (DEFAULT_TABLE table=default.Table_0e87fae7410549a88fc45e95876f13ea [id=f2d54ae909294290b3a6ad748494411a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.379984 2224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba7031c2e574617b88a597357b873a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.383484 1962 tablet_service.cc:1505] Processing CreateTablet for tablet 4e8757557bd8493f97d1a75e4e2efed9 (DEFAULT_TABLE table=default.Table_4ad9ab3f7c6c4ad2b11fc443473ab58e [id=fd51d90186ae43279185c012b888185a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.384222 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79c18ebf56e045b6a52fae8871a9107f" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:53.384423 2109 raft_consensus.cc:3060] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.384516 2231 tablet_service.cc:1505] Processing CreateTablet for tablet cfff2174bac74ab0999215c2aed5b955 (DEFAULT_TABLE table=default.Table_39cd69856c0a4284b604da1e447e172a [id=1e8834dad64e4a2f9266f10948603710]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.384850 2231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfff2174bac74ab0999215c2aed5b955. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.386292 2088 tablet_service.cc:1505] Processing CreateTablet for tablet 4e8757557bd8493f97d1a75e4e2efed9 (DEFAULT_TABLE table=default.Table_4ad9ab3f7c6c4ad2b11fc443473ab58e [id=fd51d90186ae43279185c012b888185a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.387318 2109 raft_consensus.cc:2468] T 79c18ebf56e045b6a52fae8871a9107f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
W20251026 02:56:53.387760 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 64326.9us
I20251026 02:56:53.387902 1959 tablet_service.cc:1505] Processing CreateTablet for tablet a14c0aa4fc754995acc2177f5dd3ce53 (DEFAULT_TABLE table=default.Table_3e17f384354f44e48cdfa79c4cac560d [id=5f60e15224744b6fb813266c8e8666e5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:53.387979 2053 leader_election.cc:343] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.388079 2053 leader_election.cc:304] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48; no voters: 696476265495492790739ec126a08539, 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.388331 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d06944f71fd4e42a6ea46969c86e9f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.388607 2053 leader_election.cc:304] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539; no voters: 7fef01f4f2b843538ea245a09dd087a8
I20251026 02:56:53.388713 2238 tablet_service.cc:1505] Processing CreateTablet for tablet 37a34c28786c4cebb741f8218a72502b (DEFAULT_TABLE table=default.Table_fdaf051a8a4b4899a92c293e6428e200 [id=129a0148a075452aa137277489f0ac10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.389066 2238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37a34c28786c4cebb741f8218a72502b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.389055 4494 raft_consensus.cc:2749] T 3aa7a90a29ec430286786bf8aa9d4c02 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 02:56:53.389456 4494 raft_consensus.cc:2804] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 02:56:53.389573 4494 raft_consensus.cc:697] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Running, Role: LEADER
I20251026 02:56:53.389792 4494 consensus_queue.cc:237] T 09f5265d85c6496d85c555b2b847373a P 0b6b0a6c4e144982b67980118cddba48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.391201 2375 tablet_bootstrap.cc:492] T ceb7249968ff44c1b5be1d1d18605ffe P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:53.392343 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet 72e4a4ea953f4f2786a39c0ddbd11219 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.392530 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet a02c9d84f0ec498e896ec97a75e3c154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.392853 2375 tablet_bootstrap.cc:654] T ceb7249968ff44c1b5be1d1d18605ffe P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
W20251026 02:56:53.392917 1915 leader_election.cc:343] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: d43b6f22dd4f4f9c9657f21514550312
I20251026 02:56:53.393050 1915 leader_election.cc:304] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:53.394134 4458 raft_consensus.cc:2749] T d43b6f22dd4f4f9c9657f21514550312 P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 02:56:53.396522 1657 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: Sending DeleteTablet for 3 replicas of tablet 260c6153465e48cb9b4fd24b4f7f7a83
I20251026 02:56:53.396577 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ad6906bb58a4d5fbbb64c5bc6acef8a with delete_type TABLET_DATA_DELETED (replaced by tablet 5c9addce32ec451484003598c4f4592e) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.289883 2376 ts_tablet_manager.cc:1929] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:53.296293 1975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31b6e08597bd450eb1875201cf594a82" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:53.296188 1966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "885c10070d6942f7abe52c4558d887e6" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8" is_pre_election: true
I20251026 02:56:53.399849 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 260c6153465e48cb9b4fd24b4f7f7a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:43402
W20251026 02:56:53.400638 2194 leader_election.cc:343] T 31b6e08597bd450eb1875201cf594a82 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
W20251026 02:56:53.400884 2194 leader_election.cc:343] T 885c10070d6942f7abe52c4558d887e6 P 696476265495492790739ec126a08539 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.401014 1663 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_9c1813a4f5414f74a27525b8906a380b" } new_table_name: "default.Table_f3a0788b532f4e7d9cb5f93b376ac0f5" modify_external_catalogs: true
I20251026 02:56:53.401172 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet 260c6153465e48cb9b4fd24b4f7f7a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:53.402498 1654 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_55e6f4711fcc48eda34b1f7b62a7ad2c" } new_table_name: "default.Table_f3d5635a1979472794d958110c27a278" modify_external_catalogs: true
I20251026 02:56:53.403653 1660 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b52a73c60cda4bc58b800957c4f9bed1" } modify_external_catalogs: true
I20251026 02:56:53.403816 1660 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39910:
table { table_name: "default.Table_b52a73c60cda4bc58b800957c4f9bed1" } modify_external_catalogs: true
I20251026 02:56:53.399194 2376 log.cc:1199] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-0/wal/wals/0cf76ef8f6af445eb0ce15c8d79136c5
I20251026 02:56:53.405961 2376 ts_tablet_manager.cc:1950] T 0cf76ef8f6af445eb0ce15c8d79136c5 P 7fef01f4f2b843538ea245a09dd087a8: Deleting consensus metadata
I20251026 02:56:53.406324 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1092) took 4414 ms. Trace:
I20251026 02:56:53.406416 2376 rpcz_store.cc:276] 1026 02:56:48.991438 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:48.991481 (+ 43us) service_pool.cc:225] Handling call
1026 02:56:53.143400 (+4151919us) ts_tablet_manager.cc:1151] Deleting tablet 0cf76ef8f6af445eb0ce15c8d79136c5
1026 02:56:53.143411 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:53.220775 (+ 77364us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:53.289749 (+ 68974us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:53.406287 (+116538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4151764}
I20251026 02:56:53.406857 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58894 (request call id 1087) took 4431 ms. Trace:
I20251026 02:56:53.406939 2376 rpcz_store.cc:276] 1026 02:56:48.975511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:49.016984 (+ 41473us) service_pool.cc:225] Handling call
1026 02:56:53.406719 (+4389735us) ts_tablet_manager.cc:1151] Deleting tablet cedff3287c7b4a07a81462e8c4b44970
1026 02:56:53.406729 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:53.406828 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4389468}
I20251026 02:56:53.407197 2376 tablet_replica.cc:333] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8: stopping tablet replica
I20251026 02:56:53.407398 2376 raft_consensus.cc:2243] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:53.407410 1662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_10753545f84d450b96e6008304c66a84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:53.407580 2376 raft_consensus.cc:2272] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 02:56:53.407903 1662 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10753545f84d450b96e6008304c66a84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.408140 2376 ts_tablet_manager.cc:1916] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 02:56:53.408313 1629 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.150.1:37563, user_credentials={real_user=slave}} blocked reactor thread for 99095.3us
I20251026 02:56:53.409009 1647 catalog_manager.cc:4985] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): tablet 0cf76ef8f6af445eb0ce15c8d79136c5 (table default.Table_72ee520a0e4d45fb920d1e3cc7a426b3 [id=33334e177bb94b3e9de2a7e7ee59979b]) successfully deleted
W20251026 02:56:53.409256 1647 catalog_manager.cc:4965] TS 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): delete failed for tablet cedff3287c7b4a07a81462e8c4b44970 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cedff3287c7b4a07a81462e8c4b44970 already in progress: creating tablet
I20251026 02:56:53.409520 1655 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 209968db4dee47c49497a66c3ee750b7: default.Table_907991c2b0034c118065ead079f273f9 [id=a8d490f3dbc948d38c54d14e178b736e] alter complete (version 1)
I20251026 02:56:53.410487 1658 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_971889837f9f46b29651dfe2cddc1ea5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:53.410701 1956 tablet_service.cc:1552] Processing DeleteTablet for tablet dfd3e23d83a84648bd11cd258377aecb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
I20251026 02:56:53.410710 1947 tablet_service.cc:1552] Processing DeleteTablet for tablet 209fa71af13142a58efb9f3dab3eecd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:58894
W20251026 02:56:53.411012 1658 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_971889837f9f46b29651dfe2cddc1ea5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.411731 4460 raft_consensus.cc:493] T 85420a102dae4269811dbdc64aebb20a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 02:56:53.411887 4460 raft_consensus.cc:515] T 85420a102dae4269811dbdc64aebb20a P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.412575 4460 leader_election.cc:290] T 85420a102dae4269811dbdc64aebb20a P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611), 696476265495492790739ec126a08539 (127.1.150.3:38107)
I20251026 02:56:53.412649 2098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b2eeedec7f48088b54d79394432dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.412907 1657 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_dc3c5c2144fb40f980412f4557936f87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:53.413254 2372 ts_tablet_manager.cc:1929] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 02:56:53.413384 2372 log.cc:1199] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-1/wal/wals/b2bfd9b55c484ef8a9c6345d56f794fb
W20251026 02:56:53.413374 1657 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc3c5c2144fb40f980412f4557936f87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.413780 2372 ts_tablet_manager.cc:1950] T b2bfd9b55c484ef8a9c6345d56f794fb P 0b6b0a6c4e144982b67980118cddba48: Deleting consensus metadata
I20251026 02:56:53.415210 2088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e8757557bd8493f97d1a75e4e2efed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.418298 2090 tablet_service.cc:1552] Processing DeleteTablet for tablet ded06ba1f80c4433ae78687e5990edce with delete_type TABLET_DATA_DELETED (replaced by tablet 3131bc2948e34c25a1bbda90f8dabeb3) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:53.419792 2091 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b49c0996c4d41cbaddcdf0b75846bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:30 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:53.419999 2085 tablet_service.cc:1552] Processing DeleteTablet for tablet 3aa7a90a29ec430286786bf8aa9d4c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:37 UTC) from {username='slave'} at 127.0.0.1:43402
I20251026 02:56:53.421249 2085 tablet_service.cc:1505] Processing CreateTablet for tablet a14c0aa4fc754995acc2177f5dd3ce53 (DEFAULT_TABLE table=default.Table_3e17f384354f44e48cdfa79c4cac560d [id=5f60e15224744b6fb813266c8e8666e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.421650 2085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a14c0aa4fc754995acc2177f5dd3ce53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.424463 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfff2174bac74ab0999215c2aed5b955. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.424911 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e8757557bd8493f97d1a75e4e2efed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.428488 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a14c0aa4fc754995acc2177f5dd3ce53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.428576 2091 tablet_service.cc:1505] Processing CreateTablet for tablet e1347b333f0a4582a069c12801e155c7 (DEFAULT_TABLE table=default.Table_064d2f38544443f4aee739f470193114 [id=81bdfded87ca4733863fde98ba33aa18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.428970 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1347b333f0a4582a069c12801e155c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.430557 2090 tablet_service.cc:1505] Processing CreateTablet for tablet 15d34736760b4aba8f9a19cf7aea6a4e (DEFAULT_TABLE table=default.Table_22fd677885264f808e07ce3b8ca2c5c0 [id=410566613f8c4355af5053cfaa85d09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.430923 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d34736760b4aba8f9a19cf7aea6a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.432371 1944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37a34c28786c4cebb741f8218a72502b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.436455 1957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b2eeedec7f48088b54d79394432dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.436743 2086 tablet_service.cc:1505] Processing CreateTablet for tablet daf913e05ea140d0b4cabad7104b1305 (DEFAULT_TABLE table=default.Table_814f1eb925e748a6a60c9e85eed9667f [id=052e08d5483843428198dead6408a2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.436867 2100 tablet_service.cc:1505] Processing CreateTablet for tablet 7b37023027634ae3b1b6c245cb1f2f9a (DEFAULT_TABLE table=default.Table_43fe592c9103480e8fb16e77712556d8 [id=47e347b867814bc790bfcba392276a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.437116 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf913e05ea140d0b4cabad7104b1305. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.437629 2100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b37023027634ae3b1b6c245cb1f2f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.440414 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba7031c2e574617b88a597357b873a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.443567 1947 tablet_service.cc:1505] Processing CreateTablet for tablet e1347b333f0a4582a069c12801e155c7 (DEFAULT_TABLE table=default.Table_064d2f38544443f4aee739f470193114 [id=81bdfded87ca4733863fde98ba33aa18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.443727 1953 tablet_service.cc:1505] Processing CreateTablet for tablet daf913e05ea140d0b4cabad7104b1305 (DEFAULT_TABLE table=default.Table_814f1eb925e748a6a60c9e85eed9667f [id=052e08d5483843428198dead6408a2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.444002 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1347b333f0a4582a069c12801e155c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.444239 1951 tablet_service.cc:1505] Processing CreateTablet for tablet 7b37023027634ae3b1b6c245cb1f2f9a (DEFAULT_TABLE table=default.Table_43fe592c9103480e8fb16e77712556d8 [id=47e347b867814bc790bfcba392276a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.444566 1951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b37023027634ae3b1b6c245cb1f2f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.444955 2109 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8f0124f125c40d291c2211fd95c71c5" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48"
I20251026 02:56:53.445149 2109 raft_consensus.cc:3060] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 02:56:53.446765 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 3aea01a248b345cf81d4d9862744eb78 (table default.Table_d68a91a3695240a39e5526b9cff6f6f2 [id=74d6732406a448e49911361bdc64417a]). Expected version 2 got 0
I20251026 02:56:53.446995 1661 catalog_manager.cc:5741] TS 696476265495492790739ec126a08539 (127.1.150.3:38107) does not have the latest schema for tablet 24449cb9121642009dec41220e27b193 (table default.Table_12cdd920c7d54ce7818f9fa6d5fce036 [id=e8ca26900f06405d904f48d7387baf16]). Expected version 1 got 0
I20251026 02:56:53.447710 2109 raft_consensus.cc:2468] T f8f0124f125c40d291c2211fd95c71c5 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 1.
I20251026 02:56:53.448596 1956 tablet_service.cc:1505] Processing CreateTablet for tablet 15d34736760b4aba8f9a19cf7aea6a4e (DEFAULT_TABLE table=default.Table_22fd677885264f808e07ce3b8ca2c5c0 [id=410566613f8c4355af5053cfaa85d09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.448717 2381 tablet_bootstrap.cc:492] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 696476265495492790739ec126a08539: No bootstrap required, opened a new log
I20251026 02:56:53.448844 2381 ts_tablet_manager.cc:1403] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 696476265495492790739ec126a08539: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.000s
I20251026 02:56:53.448966 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d34736760b4aba8f9a19cf7aea6a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.449513 2381 raft_consensus.cc:359] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.449679 2381 raft_consensus.cc:385] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:53.449723 2381 raft_consensus.cc:740] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 696476265495492790739ec126a08539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696476265495492790739ec126a08539, State: Initialized, Role: FOLLOWER
I20251026 02:56:53.449923 2381 consensus_queue.cc:260] T 6ce4092b7ccc4888b5cbe3b38bee2e36 P 696476265495492790739ec126a08539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
W20251026 02:56:53.450135 1659 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70d9e78abcec4a609cf20b4f79730366 consensus state has no leader
I20251026 02:56:53.452067 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf913e05ea140d0b4cabad7104b1305. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.452101 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85420a102dae4269811dbdc64aebb20a" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696476265495492790739ec126a08539" is_pre_election: true
I20251026 02:56:53.453361 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 885c10070d6942f7abe52c4558d887e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 73f0b1443e2c4294a222e62433f3b435) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.453639 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 01b2eeedec7f48088b54d79394432dd7 (DEFAULT_TABLE table=default.Table_34885bd9ba024c96a4bea0c2fedcfb4b [id=94a166ea5a59425797bb4af071a97abf]), partition=RANGE (key) PARTITION UNBOUNDED
W20251026 02:56:53.453768 1915 leader_election.cc:343] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696476265495492790739ec126a08539 (127.1.150.3:38107): Not found: Tablet not found: 95c9fbae62db4850b0f02a2443a3432b
I20251026 02:56:53.453907 1915 leader_election.cc:304] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fef01f4f2b843538ea245a09dd087a8; no voters: 0b6b0a6c4e144982b67980118cddba48, 696476265495492790739ec126a08539
I20251026 02:56:53.453974 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b2eeedec7f48088b54d79394432dd7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:53.454099 1915 raft_consensus.cc:2686] T 95c9fbae62db4850b0f02a2443a3432b P 7fef01f4f2b843538ea245a09dd087a8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251026 02:56:53.454557 2232 tablet_service.cc:1505] Processing CreateTablet for tablet 4e8757557bd8493f97d1a75e4e2efed9 (DEFAULT_TABLE table=default.Table_4ad9ab3f7c6c4ad2b11fc443473ab58e [id=fd51d90186ae43279185c012b888185a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.454877 2232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e8757557bd8493f97d1a75e4e2efed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.455837 2086 tablet_service.cc:1505] Processing CreateTablet for tablet 5cfc5d83105c492185d1560db94f8bc3 (DEFAULT_TABLE table=default.Table_379aca7c219b4cbcbdba213c3ade48d9 [id=206b71d2c27f4e988395bc0e7a44ff52]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.456344 2086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cfc5d83105c492185d1560db94f8bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.456511 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1095) took 5997 ms. Trace:
I20251026 02:56:53.456615 2372 rpcz_store.cc:276] 1026 02:56:47.458909 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.458978 (+ 69us) service_pool.cc:225] Handling call
1026 02:56:53.153001 (+5694023us) ts_tablet_manager.cc:1151] Deleting tablet b2bfd9b55c484ef8a9c6345d56f794fb
1026 02:56:53.153011 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:53.257404 (+104393us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:53.413118 (+155714us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:53.456458 (+ 43340us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5693822}
I20251026 02:56:53.456990 1648 catalog_manager.cc:4985] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): tablet b2bfd9b55c484ef8a9c6345d56f794fb (table default.Table_30e98b21e65d443097e2fc01f96f4780 [id=0d9be16d1aec415a8f9a2433f8d7ac6b]) successfully deleted
I20251026 02:56:53.457144 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1096) took 5976 ms. Trace:
I20251026 02:56:53.457238 2372 rpcz_store.cc:276] 1026 02:56:47.480487 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.480789 (+ 302us) service_pool.cc:225] Handling call
1026 02:56:53.457006 (+5976217us) ts_tablet_manager.cc:1151] Deleting tablet 97aa7f0266854c0a8f17e6bceac0c9b5
1026 02:56:53.457018 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:53.457111 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5976026}
W20251026 02:56:53.457785 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet 97aa7f0266854c0a8f17e6bceac0c9b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97aa7f0266854c0a8f17e6bceac0c9b5 already in progress: creating tablet
I20251026 02:56:53.463585 2377 ts_tablet_manager.cc:1929] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251026 02:56:53.465430 2053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.150.3:38107, user_credentials={real_user=slave}} blocked reactor thread for 76832.4us
I20251026 02:56:53.466323 1966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c966cae6886048a2b6ad3b8bd7007bcb" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
I20251026 02:56:53.467128 1966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db94738da3554fbda64c23fde3ef6c1b" candidate_uuid: "0b6b0a6c4e144982b67980118cddba48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef01f4f2b843538ea245a09dd087a8"
W20251026 02:56:53.469340 2053 leader_election.cc:343] T c966cae6886048a2b6ad3b8bd7007bcb P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.469619 4517 raft_consensus.cc:493] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 696476265495492790739ec126a08539)
I20251026 02:56:53.469738 4517 raft_consensus.cc:515] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.470285 4517 leader_election.cc:290] T 5a00bc04b3654f90af342986751344e6 P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563), 696476265495492790739ec126a08539 (127.1.150.3:38107)
W20251026 02:56:53.470736 2053 leader_election.cc:343] T db94738da3554fbda64c23fde3ef6c1b P 0b6b0a6c4e144982b67980118cddba48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fef01f4f2b843538ea245a09dd087a8 (127.1.150.1:37563): Illegal state: must be running to vote when last-logged opid is not known
I20251026 02:56:53.473580 2375 tablet_bootstrap.cc:492] T ceb7249968ff44c1b5be1d1d18605ffe P 0b6b0a6c4e144982b67980118cddba48: No bootstrap required, opened a new log
I20251026 02:56:53.473728 2375 ts_tablet_manager.cc:1403] T ceb7249968ff44c1b5be1d1d18605ffe P 0b6b0a6c4e144982b67980118cddba48: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.000s
I20251026 02:56:53.474666 2375 raft_consensus.cc:359] T ceb7249968ff44c1b5be1d1d18605ffe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.474853 2116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31b6e08597bd450eb1875201cf594a82" candidate_uuid: "696476265495492790739ec126a08539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:53.475047 2116 raft_consensus.cc:2468] T 31b6e08597bd450eb1875201cf594a82 P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696476265495492790739ec126a08539 in term 0.
I20251026 02:56:53.476022 2117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85420a102dae4269811dbdc64aebb20a" candidate_uuid: "7fef01f4f2b843538ea245a09dd087a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6b0a6c4e144982b67980118cddba48" is_pre_election: true
I20251026 02:56:53.479575 1659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_78765161758f41589130c5b262d870dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 02:56:53.480427 1655 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39910:
name: "default.Table_34003a15df6c4f31908611198dae5bf7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 02:56:53.480820 1655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34003a15df6c4f31908611198dae5bf7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.481016 1947 tablet_service.cc:1505] Processing CreateTablet for tablet 5cfc5d83105c492185d1560db94f8bc3 (DEFAULT_TABLE table=default.Table_379aca7c219b4cbcbdba213c3ade48d9 [id=206b71d2c27f4e988395bc0e7a44ff52]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.481470 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cfc5d83105c492185d1560db94f8bc3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 02:56:53.480176 1659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78765161758f41589130c5b262d870dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 02:56:53.474850 2375 raft_consensus.cc:385] T ceb7249968ff44c1b5be1d1d18605ffe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:53.492939 2375 raft_consensus.cc:740] T ceb7249968ff44c1b5be1d1d18605ffe P 0b6b0a6c4e144982b67980118cddba48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6b0a6c4e144982b67980118cddba48, State: Initialized, Role: FOLLOWER
I20251026 02:56:53.493196 2375 consensus_queue.cc:260] T ceb7249968ff44c1b5be1d1d18605ffe P 0b6b0a6c4e144982b67980118cddba48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.494288 2375 ts_tablet_manager.cc:1434] T ceb7249968ff44c1b5be1d1d18605ffe P 0b6b0a6c4e144982b67980118cddba48: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251026 02:56:53.494871 2375 tablet_bootstrap.cc:492] T 1406c22ee4ee4f94acecad9546208e79 P 0b6b0a6c4e144982b67980118cddba48: Bootstrap starting.
I20251026 02:56:53.496706 2375 tablet_bootstrap.cc:654] T 1406c22ee4ee4f94acecad9546208e79 P 0b6b0a6c4e144982b67980118cddba48: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:53.502555 2378 tablet_bootstrap.cc:492] T 3ac38e6257e74367b06b15fc49d2af81 P 7fef01f4f2b843538ea245a09dd087a8: No bootstrap required, opened a new log
I20251026 02:56:53.502864 2378 ts_tablet_manager.cc:1403] T 3ac38e6257e74367b06b15fc49d2af81 P 7fef01f4f2b843538ea245a09dd087a8: Time spent bootstrapping tablet: real 0.225s user 0.009s sys 0.005s
I20251026 02:56:53.503665 2378 raft_consensus.cc:359] T 3ac38e6257e74367b06b15fc49d2af81 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.503932 2378 raft_consensus.cc:385] T 3ac38e6257e74367b06b15fc49d2af81 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 02:56:53.504050 2378 raft_consensus.cc:740] T 3ac38e6257e74367b06b15fc49d2af81 P 7fef01f4f2b843538ea245a09dd087a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef01f4f2b843538ea245a09dd087a8, State: Initialized, Role: FOLLOWER
I20251026 02:56:53.504333 2378 consensus_queue.cc:260] T 3ac38e6257e74367b06b15fc49d2af81 P 7fef01f4f2b843538ea245a09dd087a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fef01f4f2b843538ea245a09dd087a8" member_type: VOTER last_known_addr { host: "127.1.150.1" port: 37563 } } peers { permanent_uuid: "0b6b0a6c4e144982b67980118cddba48" member_type: VOTER last_known_addr { host: "127.1.150.2" port: 44611 } } peers { permanent_uuid: "696476265495492790739ec126a08539" member_type: VOTER last_known_addr { host: "127.1.150.3" port: 38107 } }
I20251026 02:56:53.505036 2378 ts_tablet_manager.cc:1434] T 3ac38e6257e74367b06b15fc49d2af81 P 7fef01f4f2b843538ea245a09dd087a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 02:56:53.505359 2378 tablet_bootstrap.cc:492] T b8fe7825fa8d46669772cf84bc4b3afc P 7fef01f4f2b843538ea245a09dd087a8: Bootstrap starting.
I20251026 02:56:53.507293 2378 tablet_bootstrap.cc:654] T b8fe7825fa8d46669772cf84bc4b3afc P 7fef01f4f2b843538ea245a09dd087a8: Neither blocks nor log segments found. Creating new log.
I20251026 02:56:53.512010 2372 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43402 (request call id 1097) took 6015 ms. Trace:
I20251026 02:56:53.512290 2372 rpcz_store.cc:276] 1026 02:56:47.496363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:47.496815 (+ 452us) service_pool.cc:225] Handling call
1026 02:56:53.511762 (+6014947us) ts_tablet_manager.cc:1151] Deleting tablet f840050f7e314d74ac6e604354a7d4c9
1026 02:56:53.511779 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:53.511948 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6014764}
W20251026 02:56:53.512985 1648 catalog_manager.cc:4965] TS 0b6b0a6c4e144982b67980118cddba48 (127.1.150.2:44611): delete failed for tablet f840050f7e314d74ac6e604354a7d4c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f840050f7e314d74ac6e604354a7d4c9 already in progress: creating tablet
I20251026 02:56:53.513046 2372 tablet_replica.cc:333] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48: stopping tablet replica
I20251026 02:56:53.513352 2372 raft_consensus.cc:2243] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 02:56:53.513746 2372 raft_consensus.cc:2272] T ce1ab21ddf8e45bc9063b63db9c1b97f P 0b6b0a6c4e144982b67980118cddba48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:53.508705 2239 tablet_service.cc:1552] Processing DeleteTablet for tablet 892a5fab2af1432eb31d92d9859ea2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:33 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.508926 2228 tablet_service.cc:1552] Processing DeleteTablet for tablet 260c6153465e48cb9b4fd24b4f7f7a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:53 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.509128 2238 tablet_service.cc:1552] Processing DeleteTablet for tablet 24f16e40a30c41788a6f9813c235c5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:35 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.509244 2242 tablet_service.cc:1552] Processing DeleteTablet for tablet 79c18ebf56e045b6a52fae8871a9107f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:29 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.524459 2377 log.cc:1199] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539: Deleting WAL directory at /tmp/dist-test-taskiHIxsS/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761447376794369-1624-0/minicluster-data/ts-2/wal/wals/6699b50d87964f6a89c365171253ad25
I20251026 02:56:53.509349 2234 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ac38e6257e74367b06b15fc49d2af81 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c148c74fbcd4d30b9ba0ea059284427) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.509531 2226 tablet_service.cc:1552] Processing DeleteTablet for tablet a53f25688e3b45dca68984e2c543d094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:28 UTC) from {username='slave'} at 127.0.0.1:35450
I20251026 02:56:53.509653 2229 tablet_service.cc:1505] Processing CreateTablet for tablet 15d34736760b4aba8f9a19cf7aea6a4e (DEFAULT_TABLE table=default.Table_22fd677885264f808e07ce3b8ca2c5c0 [id=410566613f8c4355af5053cfaa85d09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.524879 2377 ts_tablet_manager.cc:1950] T 6699b50d87964f6a89c365171253ad25 P 696476265495492790739ec126a08539: Deleting consensus metadata
I20251026 02:56:53.524927 2229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d34736760b4aba8f9a19cf7aea6a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.525660 2377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35450 (request call id 1180) took 4247 ms. Trace:
I20251026 02:56:53.525743 2377 rpcz_store.cc:276] 1026 02:56:49.278017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 02:56:49.278257 (+ 240us) service_pool.cc:225] Handling call
1026 02:56:53.235789 (+3957532us) ts_tablet_manager.cc:1151] Deleting tablet 6699b50d87964f6a89c365171253ad25
1026 02:56:53.235804 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1026 02:56:53.350815 (+115011us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:53.463487 (+112672us) tablet_metadata.cc:641] Metadata flushed
1026 02:56:53.525638 (+ 62151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3915604}
I20251026 02:56:53.526032 2377 tablet_replica.cc:333] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539: stopping tablet replica
I20251026 02:56:53.526186 2377 raft_consensus.cc:2243] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 1 LEADER]: Raft consensus shutting down.
I20251026 02:56:53.526522 2377 raft_consensus.cc:2272] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 02:56:53.527025 2377 ts_tablet_manager.cc:1916] T 209fa71af13142a58efb9f3dab3eecd3 P 696476265495492790739ec126a08539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 02:56:53.509717 2227 tablet_service.cc:1505] Processing CreateTablet for tablet a14c0aa4fc754995acc2177f5dd3ce53 (DEFAULT_TABLE table=default.Table_3e17f384354f44e48cdfa79c4cac560d [id=5f60e15224744b6fb813266c8e8666e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.528383 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a14c0aa4fc754995acc2177f5dd3ce53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 02:56:53.509955 2230 tablet_service.cc:1505] Processing CreateTablet for tablet e1347b333f0a4582a069c12801e155c7 (DEFAULT_TABLE table=default.Table_064d2f38544443f4aee739f470193114 [id=81bdfded87ca4733863fde98ba33aa18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.510105 2231 tablet_service.cc:1505] Processing CreateTablet for tablet daf913e05ea140d0b4cabad7104b1305 (DEFAULT_TABLE table=default.Table_814f1eb925e748a6a60c9e85eed9667f [id=052e08d5483843428198dead6408a2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 02:56:53.510217 2232 tablet_service.cc:1552] Processing DeleteTablet for tablet 81775055d98349f696cdab6171ffd58b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 02:56:36 UTC) from {username='slave'